PSY 395 Design Lab 1
PSY 395 Lab 10: Deciding on a Design
How do you get from your research question to a research design?
Just guess?
Hope it comes to you in a dream?
Take what you know from 395 and make educated guesses? YES!
You need to think about your research question and the independent variable to begin:
Can you randomly assign on all variables (all of your Independent Variables)?
YesNo
The answer to this question limits the type of design you have. Why?
If you answer yes, you have true experimental design. If you answer no, you have either a quasi-experimental or an observational design.
To decide whether you have a quasi-experiment or observational experiment, you need to ask…
Can I manipulate any variables?
What does knowing about manipulation tell us?
YesNo
If yes, then you have a quasi-experimental design
If no, then you have an observational design.
Non-experimental/Observational research falls into 3 categories:
- Archival research (analyzing data that someone else has already collected)
- Survey research (collecting data on questions of interest to you)
- Correlational research – the overarching category of observational design – using correlations to test the strength of relationship between two variables
Let’s stick with true experiments for now…
The next question you want to ask is:
Do you have a pretest?
YESNO
You always assume that you have a post-test measure, so you aren't going to be asking that as one of your questions, but you do need to know if you’ll have a pre-test.
If you do have a pretest, then you have what is called a repeated measures design. Because you’ll be comparing people’s scores on the pretest with scores on the post-test to check for differences due to the treatment, you have multiple measures, which is why this design is called repeated measures. If you have multiple groups you’re comparing pretests and post-tests on, this type of design is called between groups repeated measures, because you want to see if there are differences between groups on your measures.
No pretest?
If you do not have a pretest, you have something called a randomized two-group design. While you have randomly assigned people to your conditions, all you’re looking at are the differences between groups on your post-test, because you have no pretest measure to compare people on.
Continuing on with repeated measures…
The next question you want to ask is:
Are you comparing differences across more than 2 groups?
YESNO
Why does the number of groups make a difference?
Think about a design you talked about in class that used more than 2 groups. What was unique about that design?
Not more than 2 groups?
If you do not have more than 2 groups, then you have a design called a pretest-posttest two group design. This design is just what it says: it uses a pretest and posttest for two groups (still random assignment and repeated measures) and compares differences on your DV across the two groups.
If you do have more than 2 groups, and only some of those groups take a pretest, then…
You have what is called a Solomon Four-Group Design. In this design, two groups take a pretest and the posttest, while the other two groups do not take the pretest, but only take the posttest. What advantages does this kind of design give to the experimenter (think validity)?
And that gets us to the end of true experimental designs – let’s go back to look at some quasi-experimental designs.
Remember what defines a quasi-experiment: If you cannot randomly assign on all of your variables, but you can manipulate some, then you have a quasi-experiment.
Let’s ask the same questions we did to determine what type of experimental design we had:
The next question you want to ask is:
Do you have a pretest?
YESNO
Remember, when using a quasi-experimental design, you do not have control over things such as individual differences because you did not randomly assign people to your conditions. Therefore, different groups in quasi-experimental designs are often referred to as nonequivalent because differences across the two groups might exist (due to no random assignment).
If you do have a pretest…
Then you have what is called a pretest-posttest nonequivalent control group design. This too is a repeated measures design, because you have both a pretest and posttest and you have one control group and one experimental group. Your goal is to examine the differences in outcomes between your two groups due to your treatment. The only thing different about this type of design from a “true experiment repeated measures” is that your groups are not equivalent as explained just a minute ago.
If you do not have a pretest…
Then you have what is called a static-group comparisondesign. This type of design simply compares differences in your DV across two groups, without taking a pretest measure in either group. Remember, one group will still receive a treatment while the other serves as the control. You have a between groups design, however it is NOT repeated measures. What type of threat to validity does this design open you up to?
If you have multiple pretests (and multiple posttests)…
Then you have something called an interrupted time series design. This type of design uses multiple pretests prior to your treatment and multiple posttests following the treatment. What type of threat does this help the researcher understand better?
To narrow down the type of design even further, the next question you ask is “Do you have more than 2 groups?”
The design discussed in your book only talks about what happens when you have 1 and only 1 group.
This type of design is called a one-group pretest-posttest design. What kind of comparison do you think this type of design makes? (remember you’ve only got one group, so this would not be a between groups design).
This type of design is also called simple panel research – this is a within-groups (repeated-measures) design, since it looks at changes in your outcome within just one group, using the same participants prior to the treatment as after the treatment.
Non-Experimental/Observational designs are used when you have NO random assignment and NO manipulation!
3 General Categories of non-experimental design:
- Archival- using data that has already been collected by another person
- Survey-collecting data by administering questionnaires - Note: while you do not use assignment, you can/will use random or non-random sampling!
- Correlational- using correlation to test the strength of the relationship between two variables.
An important thing to remember with survey and archival data is that you can have within-group designs as well between (independent) group designs.
To review: The types of designs we’ve covered are:
Non-Experimental/Observational
-Archival research
-Survey research
-Correlational research
True Experiment
-Randomized two group design (posttest only)
-Pretest-posttest two group design (pre-post repeated measures: between groups)
-Solomon four group design
Quasi-Experiment
-Pretest-posttest nonequivalent control group design
-Static group comparison design
-Interrupted time series design
-One group pretest-posttest design (within group)
So, now you can figure out what type of design you have given some information about the experiment. This is part of your homework for today. The other part is deciding what type of statistical test you would use to test your results.
Identifying Statistical Tests
Some of this can be figured out, but some of this also requires memorization.
Let’s review: what do each of these tests tell us?
Correlation
Regression
Independent Measures t-Test
Type of Design / Statistical TestNon-experimental/Observational / Correlation
Regression
Analysis of Covariance
ANOVA
Independent Measures t-test
Repeated Measures t-test
Randomized Two Group Design / Independent Measures t-test
Pretest-Posttest Two Group Design / Repeated Measures t-test
Solomon Four Group Design / ANOVA
Pretest-Posttest Nonequivalent Control Group Design / Repeated Measures t-test
Static Group Comparison / Independent Measures t-test
Interrupted Time Series Design / Repeated Measures ANOVA
One Group Pretest-Posttest Design / Repeated Measures t-test (within group)
Repeated Measures t-test
ANOVA
Something new: Analysis of covariance (ANCOVA)
What’s the difference between ANOVA and ANCOVA?
ANCOVA acknowledges that in observational research, there may be additional variables that covary or change with the variables you’re interested in. The relationship between these additional, or confounding variables, can influence the relationship between the IV and DV that you want to look at.
ANCOVA is a technique that removes the influence of the covarying variable from the relationship between your IV and DV so that you can look at the “pure” relationship between your two variables. In other words, ANCOVA allows you to look at the strength of the relationship between your IV and DV by removing the influence of a confounding variable.
Project Questions?
Methods Section: when proposing your research design, you must go beyond saying whether you have a non-experimental, quasi-experimental or true experimental design. You need to describe a specific design (e.g., static group comparison, randomized two group design) and give enough detail so that it’s clear we understand the design you’re using and why it is appropriate for your research question.
Any other project related questions…
The next section of your project requires you to:
1)Assume that all went well and your results support your research hypothesis. What are the internal and external threats to validity that might lead you to draw the wrong conclusion about your research results? (you need to give specific examples of threats to validity here, at least one of each)
2)Assume that your results failed to support your research hypothesis. What might have gone wrong in your research to yield this result? (this is where you would discuss the limitations of your research and critique what you’ve proposed – start with your methods section if you’re having trouble)
PSY 395 Design Lab 1
Start Here: Can you randomly assign
on all variables?NO Can you manipulate NO
any IV?
Non-Experimental design
YES YES- Archival research
- Survey research
- Correlational research
True Experimental DesignQuasi-Experimental Design
Do you have
a pretest?
YES NO YES NOYES, I have multiple pretests
Repeated RandomizedPretest-Posttest Static-Group Interrupted Time Series
MeasuresTwo-GroupNonequivalent ComparisonDesign
DesignDesignControl Group Design
Do you have
more than 2
groups?
YESNONO, I only have 1 group
One-Group Pretest-Posttest Design
keep goingPretest-Posttest
Two-Group Design
Do only some
of your groups
take a pre-test? YES Solomon Four-Group Design
PSY 395 Design Lab 1
Homework for Lab #10
Read the following 3 scenarios. For each scenario, identify: (a) the design of the experiment and explain why, (b) the statistical test you would use to analyze the data, and (c) one specific validity threat (external or internal) and explain why it’s a threat.
1. Can blueberries slow down aging? A study indicates that antioxidants found in blueberries may slow down the process of aging. In this study, 19-month old rats (equivalent to 60-year old humans) were fed either their standard diet or a diet supplemented by either blueberry, strawberry, or spinach powder. After eight weeks, the rats were given memory and motor tests. Although all supplemented rats showed improvement, those supplemented with blue berry powder showed the most notable improvement.
2. [abstract from a published study] We examined the effectiveness of a cognitive intervention to help adolescents cope with stress and other forms of negative emotional arousal. Through this intervention procedure, youths learned how to identify and monitor stress-promoting cognitions, restructure these cognitions into more adaptive thoughts, use self-instructions to control stress-engendering self-statements, and practice and apply these acquired skills. A total of 21 adolescent males, 16–17 years old, participated in the study. The project was conducted at a college preparatory, parochial high school for boys in a midwestern city. Admission to this school is based on competitive examinations taken by students from over 100 elementary schools in the surrounding area. In recent years, 98% of the graduates have enrolled in college. The nature and purpose of the project was described to juniors and seniors by the head guidance counselor, and any students who were interested in learning more about the stress-management project were asked to attend an orientation meeting. Originally, 30 youths attended this orientation meeting, during which the research was described in full by the researchers and parental and youth consent forms were distributed. Twenty-four youths returned signed consent forms and volunteered for the project. These youths were randomly assigned to either the experimental group that received training or a waiting list control group. Before the training was completed, three youths in the experimental group dropped out of the study because of schedule conflicts. As a result, the final sample was composed of 9 experimental youths and 12 control youths. All youths were White with the exception of one Black youth in the experimental group. Both groups were given pre- and post-assessments in 1-hr group sessions on the outcomes of interest.
3. A developmental psychologist observed children who attended daycare at least 30 hours per week and compared their behavior to their peers being raised at home. The psychologist found that the daycare children scored significantly higher than the other children on a measure of aggression.