Research Methods Midterm 3
Name and describe a procedure for controlling for possible order effects in an experiment.
- COUNTERBALANCING the order in which conditions are presented. - COMPLETE COUNTERBALANCING = all possible orders are included in the experiment, with equal numbers of people in each. - Counterbalancing when there are too many possible orders: - LATIN SQUARE = a limited set of orders that ensures that 1) each condition goes first, and last, and all the other positions, the same number of times, and 2) each condition precedes and follows each other condition once.
Give examples of cross-sectional, longitudinal, and sequential designs, and summarize the pros and cons of each type.
- CROSS-SECTIONAL: Persons of different ages are studied at only one point in time. (Eg. study people who are currently, 20, 30, and 40 years old and all given the same test and see who performed better). - more common method as it is less expensive and yields results immediately. - Disadvantages: researchers must infer that among age groups are due to the developmental variable of age. the developmental change is not observed directly among the same group of people but instead is based on comparisons among different cohorts of individuals. - LONGITUDINAL STUDY: The same group of people is observed at different points in time as they grow older. - the best way to study how scores on a variable at one age are related to another variable at a later age. - is expensive and difficult, and people may move, die, or lose interest in the study. - SEQUENTIAL DESIGN: Do a cross-sectional and longitudinal study at the same time. (Ex: Start with people who are currently 20, 30, and 40 years old. Then collect data in 10 years, when they are 30, 40, and 50 years old.) - takes fewer years and less effort than longitudinal and the researcher reaps the rewards immediately because data on the different age groups are available in the first year of the study.
In a 2 x 2 factorial design, in terms of possible outcomes: what are the three comparisons you would do/questions you would answer? (see Ch. 10 Powerpoint, Slide 12)
1. There may or may not be a significant main effect for independent variable A. 2. There may or may not be a significant main effect for independent variable B. 3. There may or may not be a significant interaction between IV A and IV B.
What is a cohort? Why is this concept important in developmental research?
A cohort is a group of people born at around the same time, exposed to the same events in society and influenced by the same demographic trends such as divorce rates, and family size. - differences among cohorts reflect different economic and political conditions in society, different music and arts, different educational systems and different child rearing practices. in a cross-sectional study, a difference among groups of different ages may reflect developmental age changes, however the differences may result from cohort effects.
Explain how an ABA, ABAB, multiple-baseline, or control series design all help rule out alternative explanations.
ABA DESIGN OR WITHDRAWL DESIGN: Record the baseline behavior, then record behavior during the treatment period, then remove to treatment to record a second baseline period. - MULTIPLE-BASELINE: the effectiveness of the treatment is demonstrated when a behavior changes only after the manipulation is introduced. in order to demonstrate the effectiveness of the treatment, such a change must be observed multiple circumstances to rule out the possibility that other events were responsible. - Gave the treatment at different locations (eg. home vs work). - CONTROL SERIES DESIGN: try to find a comparison group, to act as a control group. the fact that fatalities declined in Connecticut but not the other states suggests that the crackdown - sometimes it's unethical or impossible to reverse a treatment. In this case, you can take multiple baseline measures at different times, and multiple post-treatment measures at different times.
What are the advantages of a repeated measures design, compared with an independent groups design? What are the disadvantages?
ADVANTAGES: Each participant is their own control/comparison. - Need fewer participants since you get to use them twice. - Easier to find statistically significant differences, because you can compare each person to him/herself. DISADVANTAGES: - Need to watch out for order effects, where the previous condition affects the next one. - Practice effect: someone might perform better on the second condition because of experience on the first. - Fatigue effect: someone might do worse on the second condition because they're tired from the first. - Carryover effect: the intervention from the first condition might still be affecting someone when they do the second.
Practice working with data from a 2 x 2 factorial design, such as Outcome 1 and Outcome 2 on Handout 5 (HW on Outcomes of Factorial Designs). How can you use these numbers to determine whether there are main effects, or interactions?
Analysis of variance (ANOVA) is the statistical procedure used to assess the statistical significance of the main effects and interaction in a factorial design. You look at the simple main effects. Basically, you pretend that you had separate experiments at each level of one of the IVs. - find the average of A and average of B - if the average are the same for all of A then their is no interaction.
Explain the difference between internal and external validity. What kinds of problems in experiments are threats to each type? pg 152
INTERNAL VALIDITY: when the results of the experiment can confidently be attributed to the effect of the independent variable. Refers to the accuracy of conclusions drawn about cause and effect. EXTERNAL VALIDITY: it is the extent to which the results of a study can be generalized to and across other situations, people, stimuli, and times.
Give an example of a one-group posttest-only design, and explain why these results cannot be interpreted.
ONE-GROUP POSTTEST-ONLY DESIGN: One-group posttest-only design: (Ex: Does sitting close to a stranger cause them to move away?) - lacks a crucial element of a true experiment : a control or comparison group. there must be some sort of comparison group for you to be able to interpret the results. - so designing an internally valid experiment is hard using this method.
Describe and give examples of a one-group pretest-posttest design and a nonequivalent control group design, explaining the advantages and problems with each one.
ONE-GROUP PRETEST-POSTTEST DESIGN: measure the participants before the manipulation (a pretest) and again afterward ( a posttest). - (eg. Ex: Does a relaxation training program reduce cigarette smoking?) - A change in a one-group pretest-posttest design can't rule out alternative explanations such as: - HISTORY: Maybe some other event, such as a high-profile celebrity death, caused the reduction in smoking. - MATURATION: Maybe people become more concerned with their health as they age. - TESTING: Maybe keeping track of every cigarette caused people to smoke less. - INSTRUMENT DECAY: Maybe people got lazy about recording every cigarette over the length of the study. - REGRESSION TOWARDS THE MEAN (statistical regression): If people were selected for this program because they smoked a lot, they would tend to smoke less at the end by chance alone. - NONEQUIVELENT CONTROL GROUP DESIGN: This design does have a control group, but the participants were not randomly assigned. one of the most useful quasi-experimental designs. Not a true experimental design because assignment to groups is not random, the two groups may not be equivalent. - We have the advantage, however of knowing the pretest score. Thus we can see whether the groups were the same on the pretest. - You can also look at the change in scores from pretest to posttest, even if the groups are not equivalent. - if the IV has an effect then the experimental group should show a greater change than the control group.
Distinguish between a posttest-only, pretest-posttest, Soloman four-group, repeated measures, and matched pairs design.
POSTTEST-ONLY: (1) obtain two equivalent groups of participants, (2) manipulate the independent variable and (3) measure the effect of the independent variable on the dependent variable. choose participants randomly assign them to two groups and one is the control while the other is the experimental group. then you measure the dependent variable for both groups. PRETEST-POSTTEST DESIGN: the only difference between posttest-only and pretest-posttest design is that in the latter a pretest is given before the experimental manipulation is introduced. so you'll give all the participants a pretest to measure their baseline for the dependent variable. Then randomly assign them to two groups (one is the control while the other is the experimental group). Then you measure the dependent variable for both groups again and see if there is a change. -PROS OF PRETEST: Pretest is a good idea with small sample sizes. Pretest enables us to see a change after IV, including for individual participants. - CONS OF PRETEST: In the real world, there is usually no pretest, so having one may reduce external validity. Pretest may be time-consuming and might give away the topic of your experiment. SOLOMON FOUR GROUP DESIGN: Half the participants receive only the posttest while the other half receive both the pretest and posttest. if there is no impact of the pretest, the posttest scores will be the same in the two control groups and in the two experimental groups. Lets us see if the pretest affected the final results. REPEATED MEASURES: participants are repeatedly measured on the dependent variable after being in each condition of the experiment. Each participant is their own control/comparison. Need fewer participants since you get to use them twice. Easier to find statistically significant differences, because you can compare each person to him/herself. ADVANTAGES OF REPEATED MEASURES: (1) fewer research participants are needed, (2) they are extremely sensitive to finding statistically significant differences between groups. DISADVANTAGE OF REPEATED MEASURES DESIGN: the different conditions must be presented in a particular sequence. (1) Need to watch out for ORDER EFFECTS, where the previous condition affects the next one. - PRACTICE EFFECT: someone might perform better on the second condition because of experience on the first. - FATIGUE EFFECT: someone might do worse on the second condition because they're tired from the first. - CARRYOVER EFFECT: the intervention from the first condition might still be affecting someone when they do the second. - Deal with this problem either by delaying between conditions or by counterbalancing the order in which conditions are presented. MATCHED PAIRS DESIGN: instead of simply randomly assigning participants to groups, the goal is to first match people on a participant variable such as age or a personality trait. the matching variable will either be the dependent measure of the variable that is strongly related to the dependent variable.
Describe the difference between a straightforward and staged manipulation, using examples.
STRAIGHFORWARD MANIPULATION: Manipulate variables with instructions and stimulus presentations. stimuli may be presented verbally, in written form, via videotape, or with a computer. (eg. memory research, education programs). STAGED MANIPULATION: sometimes it is necessary to stage events during the experiment in order to manipulate the independent variable successfully. used for two reasons (1) the researcher may be trying to create some psychological state in the participants, such as frustration, anger or temporary lowering or raising of self-esteem. (2) may be necessary in order to stimulate some situation that occurs in the real world. (eg. testing a sense of entitlement, if people behaved more selfishly after being unfairly treated)
What is an IV x PV design? Be able to identify PVs in a verbal description of an experiment (such as the designs in Handout 4).
Some "independent variables" in a factorial design are not actually manipulated. These are called participant variables: (Ex: age, sex, race, diagnostic category, separated based on weight, preexisting condition etc.) - Also called subject variables or attribute variables.
What are the advantages of using a pretest-posttest design over a posttest-only design?
Some advantages to using the pretest-posttest design is that a is a good idea with small sample sizes. - Pretest enables us to see a change after IV, including for individual participants. - Especially important if some participants drop out after the start of the experiment (called attrition or mortality). - The dropouts tend to not be representative of the entire group, so losing them can distort the average result.
How are factorial designs labeled (i.e. what does a "2 x 3 factorial design" mean)?
a 2 x 3 factorial design means we are testing two independent variables with the first variable having two levels (ie. social vs unsociable) and the second variable having three (food intake low, medium, high).
What does it mean when two variables are confounded
a confounding variable is a variable that varies along with the independent variable. confounding occurs when the effects of the independent variable and an uncontrolled variable are intertwined so that you cannot determine which of the variables is responsible for the observed effect on the dependent variable.
What is a curvilinear relationship, and why is a complex experimental design necessary to identify this pattern?
a curvilinear relationship is when the direction of the relationship changes because there is a wide range of levels of the independent variable. - It's possible that if we only test two levels, we could miss a curvilinear relationship entirely! - Imagine if we only tested Level 1 and Level 3. Our data would suggest no relationship between the IV and DV. - When we also test Level 2,we see the inverted-U.
What is a confederate? Why are they used in psychological research?
an accomplice that is usually part of the staged manipulation and may be useful to create a particular social situation.
What is the purpose of a manipulation check? pg 180
an attempt to directly measure whether the independent variable manipulation has the intended effect on the participants. Make sure your IV manipulation actually worked, before you use it to find a relationship with the DV.
When given a verbal description of an experiment with more than two independent variables, be able to identify the design (ex: 3 x 2 x 3).
eg of 3x3: College sophomores were given a short course in speed reading. Three groups had courses lasting for 5, 15, or 25 sessions. At the conclusion of the course, participants were asked to read a paragraph, followed by a test of comprehension. Before taking the test, participants in each group were offered a monetary incentive: no money, $1, or $10 for a certain level of performance. The researcher collected the reading time and number of correct items on the comprehension test for each participant. - There are 9 possible conditions - The manipulated variables are the number of sessions and monetary incentives. - This is an independent variable design because the participants were not chosen based on differing characteristics. - No this is not a repeated measure design because the participants were not the same for all the groups. - The dependent variable is the test of comprehension
Explain how history, maturation, testing, and instrument decay are all potential problems for interpreting the results of a one-group pretest-posttest design.
- HISTORY: refers to any effect that occurs between the first and second measurements but is not part of the manipulation. any such event is confounded with the manipulation. (eg. Maybe some other event, such as a high-profile celebrity death, caused the reduction in smoking). - MATURATION: people change over time and any changes that occur systemically over time are called maturation effects. (eg. Maybe people become more concerned with their health as they age). - TESTING: (eg. testing becomes a problem if simply taking the pretest changes the participants behavior. (eg. Maybe keeping track of every cigarette caused people to smoke less). reduction found could be the result of taking the pretest rather than of the program itself. - INSTRUMENT DECAY: sometimes basic characteristics of the measuring instrument change over time. (eg. Maybe people got lazy about recording every cigarette over the length of the study.) - REGRESSION TOWARDS THE MEAN (statistical regression): is likely to occur whenever participants are selected because they score extremely high or low on some variable. extremely high scores are likely to get lower (closer to the mean) and extremely low scores are likely to become higher (closer to the mean). (eg. If people were selected for this program because they smoked a lot, they would tend to smoke less at the end by chance alone).
Explain how practice, fatigue, and carryover effects can interfere with results in a repeated measures design. Use examples.
- ORDER EFFECT: the order of presenting the treatment affects the dependent variable. where the previous condition affects the next one. - PRACTICE EFFECT: someone might perform better on the second condition because of experience on the first. - FATIGUE EFFECT: someone might do worse on the second condition because they're tired from the first. - CARRYOVER EFFECT: the intervention from the first condition might still be affecting someone when they do the second.
What is regression towards the mean, and why does this occur?
- REGRESSION TOWARDS THE MEAN (statistical regression): is likely to occur whenever participants are selected because they score extremely high or low on some variable. extremely high scores are likely to get lower (closer to the mean) and extremely low scores are likely to become higher (closer to the mean). (eg. If people were selected for this program because they smoked a lot, they would tend to smoke less at the end by chance alone). - Will occur whenever you gather a set of extreme scores taken at one time and compare them with scores taken at another point in time. - statistical regression also occurs when we try to explain events in the 'real world' as well. - these problems can be eliminated by using an appropriate control group. a group that does not receive the experimental treatment provides an adequate control for the effects of history, and statistical regression, and so on.
Describe and give examples of self-report, behavioral, and physiological measures of dependent variables.
- SELF-REPORT MEASURES: can be used to measure attitudes, liking for someone, judgments about someone's characteristics, intended behavior, emotional state, confidence in ones judgement. (eg. Attitudes, opinions, emotional states, etc.) - BEHAVIORAL: Are direct observations of behavior. their is an endless amount of behavior that can be measured. (eg. Whether or not a participant does something, how many times, reaction time, duration, etc.) - PHYSIOLOGICAL MEASURES: Recording body responses (heart rate, hormone secretion, GSR, EMG, EEG, MRI, fMRI, etc.)
What are some important considerations in deciding how to manipulate the independent variable in an experiment?
- STRAIGHTFORWARD MANIPULATION: Manipulate the IV by giving different instructions or materials - (Ex: Show control and experimental group different messages about reusing towels) - STAGED MANIPULATIONS: Manipulate the IV by making certain events occur - Usually to create a certain psychological state, or simulate some situation that occurs in the real world - Might involve a confederate pretending to be a regular participant, but actually following the researcher's instructions - Construct an operational definition, then - Develop a set of instructions/stimuli you can manipulate as the independent variable.
Discuss the overall dilemma regarding the strength of the independent variable manipulation. Why would you want to make the manipulation as strong as possible? Why wouldn't you?
- a strong manipulation maximizes the differences between two groups and increases the chances that the independent variable will have a statistically significant effect on the dependent variable. - In general, you want to make your manipulation as strong as possible, maximizing the IV difference between the control and experimental groups. - This is how you figure out in the early stages of research if a relationship between the IV and DV exists at all. If it does, you can try more subtle differences to pinpoint what's going on. - However, you still want to make the manipulation: - Realistic, for the sake of external validity, and - Ethical, to avoid harm to participants.
Discuss possible sources of experimenter bias. Why are these important, and how can they be avoided?
- experimenters are usually aware of the purpose of the study and thus may develop expectations about how participants should respond. - if the experimenter has a hypothesis, they may unconsciously try to "help" confirm it. They may unintentionally treat participants differently in the two conditions, or record their data in subtly biased ways. - HOW TO REDUCE IT: - Training, consistency, and automated procedures can help reduce this. - Also, a single-blind experiment (i.e. the participant doesn't know which group they're in) or a double-blind experiment (i.e. neither the participant nor the experimenter know) is a great solution.
Explain what is meant by a "ceiling" or "floor" effect in measuring the dependent variable. Describe an example of a study that might produce a ceiling effect. Give another example for a floor effect.
CEILING EFFECT: if the test is too easy, almost everyone gets 100%. IV seems to have no effect on DV because participants quickly reach the maximum performance level. (eg. - FLOOR EFFECT: if the test is too hard, almost everyone fails. - Either way, you didn't capture the full range of outcomes.
What are demand characteristics, and how can they be avoided? (For example, what is a placebo, and what is it for?)
DEMAND CHARACTERISTICS: any feature in an experiment that might inform participants of the purpose of the study. if participants know/figure out the purpose of the study, they may try to "help" confirm it. - can be avoided by trying to disguise the dependent variable by using unobstructive measures or by placing FILLER ITEMS in questionnaires. one can also ask the participants about their perception of the purpose of the research. - may be eliminated when people are not aware that an experiment is taking place or that their behavior is being observed. - If the experiment involves a treatment, a placebo group (placebo = fake treatment) can be added to rule out patient expectations for improvement being the cause of improvement.
Chapter 8
Experimental design
Describe the difference between an independent groups design and a repeated measures design, using an example.
INDEPEDNT GROUPS DESIGN: Different participants are assigned to each of the conditions using random assignment this will prevent any systematic biases and the groups will be considered equivalent in terms of participant differences. so participant differences cannot be an explanation for the results of the experiment. REPEATED MEASURES: participants are repeatedly measured on the dependent variable after being in each condition of the experiment. Each participant is their own control/comparison. Need fewer participants since you get to use them twice. Easier to find statistically significant differences, because you can compare each person to him/herself. CHOOSING BETWEEN THE TWO: An independent measures design consists of using different participants for each condition of the experiment. A repeated-measures design consists of testing the same individuals on two or more conditions.
Discuss the interrupted time series design, and explain why it is valuable for interpreting whether a particular intervention had an effect.
INTERRUPTED TIME SERIES DESIGN: - examine the traffic fatality rates over an extended period of time both before and after the reform was instituted. - this single comparison is really a one-group pretest-posttest design with all of that design's problems of internal validity. - Eg. Did a crackdown on speeding in Connecticut passed in 1955 reduce speeding? Yes, there was a big drop in 1956, but that was after a record high in 1955 (could have been regression towards the mean). - the data from years before and after the crackdown though give researchers a less ambiguous interpretation than would be possible with data for only those two years. - A CONTROL SERIES DESIGN: use a control group and compare it to your finding in the interrupted time series design. - the fact that the fatality rates in the control states remained relatively constant while those in Connecticut consistently declined led them to conclude that the crackdown did indeed have some effect.
What is a reversal design? Describe an example, and explain and what the purpose is of doing it this way.
In a reversal design, the participant is tested in a baseline condition, then tested in a treatment condition, and then returned to baseline. If the dependent variable changes with the introduction of the treatment and then changes back with the return to baseline, this provides strong evidence of a treatment effect.
State the key difference between true experimental designs and quasi-experimental designs.
In a true experiment, participants are randomly assigned to either the treatment or the control group, whereas they are not assigned randomly in a quasi-experiment. This could be because of unethical, impractical, or impossible reasons.
Imagine conducting the same experiment as an independent groups, repeated measures, or mixed factorial design. How does the decision of what design to use affect the number of participants you will need?
Independent groups design (random assignment of different people to each group). Aka "between subjects design." a different group of participants will be assigned to each of the four conditions. - Repeated measures design (use the same individuals in all conditions of the study). Aka "within subjects design." The same participants will participate in all conditions. this means you need less participants. - mixed factorial design: half are assigned to level 1 of independent variable while the other half are assigned to level 2. - Can be completely independent groups, completely repeated measures, or a mixed factorial design.
Describe the procedure for conducting a matched pairs design, and explain why you would do this.
Instead of randomly assigning participants to groups, match them into pairs on a relevant variable (ex: age, extraversion, GPA), and randomly assign one person in each pair to experimental or control group. - Helps ensure the groups are equivalent, at least on the match variable. Most important with a small sample.
What is meant by a multiple-baseline design across subjects, or across situations? Provide examples.
MULTIPLE-BASELINE ACROSS SUBJECTS: the behavior of several subjects is measured over time, for each subject, though the manipulation is introduced at a different point in time. MULTIPLE-BASELINE ACROSS SITUATIONS: In which the same behavior is measured in different settings, such as at home and at work. manipulation is introduced at a different time in each setting with the expectation that a change in behavior in each situation will occur only after the manipulation.
Be able to identify a 3 x 3, 2 x 2, or 2 x 3 factorial design from a verbal description of an experiment (such as the designs in Handout 4).
eg. 3x2 interaction: A researcher studied the influence of intensity of room illumination (low, medium, and high) on reading speed among fifth graders. Also, children were classified as "good" or "poor" readers from achievement test scores. Each group of children read 750-word passages under all three levels of illumination (three reading trials). The order of trials for each child was randomly determined. - 6 conditions - The manipulated variable is the room illumination (low, medium, and high) - This is a participant variable because the participants were split into two groups based on reading level (ie. 'good' vs 'poor' readers). - No, this is not a repeated measures design because the same individuals did not participate in all conditions. One child read the passage under all three illumination conditions, but the groups were split based on achievement scores - The dependent variable is the student's reading speed.
Define what "mortality" means in the context of experimental research, and explain why this is a problem.
some participants drop out after the start of the experiment (called attrition or mortality). the drop-out factor in experiments. -This is a problem because the dropouts tend to not be representative of the entire group, so losing them can distort the average result. mortality can be an alternative explanation for the results.
Explain the general concept of a factorial design. What is done here, and why?
used when testing more than one IV in a single experiment. You would do this if you think it's possible that the effect of one of the IVs on the DV depends on the level of the other IV. This is called an interaction.
How do you calculate the number of conditions in a factorial design (for example, how many conditions are there in a 2 x 3 x 3 design)?
you multiply all the numbers on the factorial design by each other (eg. 3x3x2 = 18 interactions)
