Chapter 10 - Quasi-Experimental Design

maturation

Change associated with the passage of time per se is called maturation. Changes participants undergo in an experiment that are due to maturation and not due to the treatment can threaten internal validity

instrumentation

Changes over time can take place not only in the participants of an experiment, but also in the instruments used to measure the participants' performance. These changes due to instrumentation can threaten internal validity if they cannot be separated from the effect of the treatment.

14. How does a non-equivalent control group help with that quest?

• A non-equivalent control group helps with that request because it can control the major threats to internal validity such as history effects and changes in measurement that occur at the same time as the treatment.

1. What are the three criteria for internal validity? Which one is most challenging for quasi-experiments?

• Three criteria • Covariation • Time-order relationship • Elimination of plausible alternative causes • Most challenging for quasi-experiments: elimination of plausible alternative causes

15. What research would you propose to provide stronger evidence for the relationship between problem solving skills and stress levels? What secondary variables would you want to control, or measure?

o Did on other exam o Roll out something beyond survey o Experimental manipulation- control group, treatment group ♣ Get measures of both ♣ Beginning both equal, intervention occurs

Be prepared to put together a quasi-experiment using one of the following designs: one group pretest-posttest design nonequivalent control group design simple interrupted time-series design time series with nonequivalent control group design

o Example of each from PowerPoints o Start with simplest one, scale it up ♣ One group pretest post-test ♣ Simple interrupted time series ♣ Time series with nonequivalent comparison group

subject attrition

A threat to internal validity occurs when participants are lost from an experiment, for example, when participants drop out of the research project. The loss of participants changes the nature of a group from that established prior to the introduction of the treatment—for example, by destroying the equivalence of groups that had been established through random assignment.

contamination:

Occurs when there is communication of information about the experiment between groups of participants

order/practice effects

Practice effects happen when a participant's performance in repeated measures designs could potentially change across conditions simply because of repeated testing- not because of independent variable, getting used to experiment, boredom, should be balanced across conditions in repeated measures designs so that practice effects "average out" across conditions (counterbalancing)

selection

Selection is a threat to internal validity when, from the outset of a study, differences exist between the kinds of individuals in one group and those in another group in the experiment.

regression

Statistical regression can occur when individuals have been selected to participate in an experiment because of their "extreme" scores. Statistical regression is a threat to internal validity because individuals selected from extreme groups would be expected to have less extreme scores on a second test (the "posttest") without any treatment simply due to statistical regression.

testing

Taking a test generally has an effect on subsequent testing. Testing can threaten internal validity if the effect of a treatment cannot be separated from the effect of testing.

history:

The occurrence of an event other than the treatment that can threaten internal validity if it produces changes in the research participants' behavior

novelty effects:

Threats to internal validity of a study that occur when people's behavior changes simply because an innovation (e.g., a treatment) produces excitement, energy, and enthusiasm; a Hawthorne effect is a special case of novelty effects.

9. What do they mean by the pre-experimental design and why does it have so little internal validity?

• A pre-experimental design is a one-group pretest-posttest design, or a bad experiment, because it has so little internal validity. It has such little internal validity because any obtained difference between the pretest and posttest scores could be due to the treatment or to any of several threats to internal validity like history, maturation, testing, instrumentation threats, experimenter expectancy effects, and novelty effects. Researchers cannot make any conclusions about the effectiveness of treatment in this bad experiment. o Pilot design, only have the treatment group o Only looking at one group before and after o Lots of things could account for this effect, practice effect maybe o Adding control group allows us to eliminate threats to internal validity

13. What do we mean by a simple interrupted time-series, and what are we looking for in these designs?

• A simple interrupted time-series design involves researchers examining a series of observations both before and after a treatment. We are looking for abrupt changes or discontinuities in the time-series data at the time that the treatment was implemented.

7. What do we mean by contamination and what are three ways it can influence independent groups?

• Contamination: Occurs when there is communication of information about the experiment between groups of participants • 3 ways it can influence independent groups o May lead to resentment (control group learns they are not receiving a desirable treatment) o Rivalry (control condition is motivated to work harder so they don't look bad compared to individuals in a treatment group) o Diffusion of treatment (control group may apply information to themselves to imitate those receiving the treatment)

5. What additional steps should we take to deal with plausible alternatives in quasi-experiments?

• Create a nonequivalent control group design, a treatment group and a comparison group are compared using pretest and posttest measures • If the two groups are similar in their pretest scores prior to treatment but differ in their posttest scores following treatment, researchers can more confidently make a claim about the effect of treatment o Testing in school or daycare or senior center o Given up some control--to gain that back ♣ Gather data on things that you think are potential confounds ♣ Unable to control kids in classroom, teacher's personality, etc. ♣ Include data on potential confounds in data analyses ♣ Measure number of different things and see if it is having an effect when controlling for these variables

1. What are the four main goals of scientific research?

• Description, prediction, explanation, application

12. What do we mean by external validity and how do we evaluate that in quasi-experimental research?

• External validity is the extent to which the findings in research can be generalized to the population. The best evidence for the external validity of research findings is replication with different populations, settings and times. • Through repetition- effect in one group-->apply to another group

10. What do we mean by a non-equivalent control group and what are the advantages to having one?

• In a non-equivalent control group design, a treatment group and a comparison group are compared using pretest and posttest measures. The advantages to having one is that researchers can more confidently make a claim about the effect of treatment if the two groups are similar in their pretest scores prior to treatment but differ in their posttest scores following treatment. Threats to internal validity like history, maturation, testing, instrumentation, and regression can be controlled in this type of design.

8. What are the four main differences between true and quasi-experimental designs?

• Quasi-experiments provide an important alternative when true experiments are not possible. • They also lack the degree of control in true experiments, specifically they lack random assignment. • Researchers also much seek additional evidence to eliminate threats to internal validity when they do quasi-experiments rather than true-experiments. • A one-group pretest-posttest design is called a pre-experimental design because it has such little internal validity. o No random assignment in quasi o Lack of control of external events o Harder to eliminate plausible alternatives o Able to ask additional questions in quasi experimental designs ♣ Wider variety of questions tap into experimental groups

1. Which characteristic is often missing in quasi-experimental designs?

• Quasi-experiments typically lack random assignment o Random assignment o Unable to randomly assign to 2 groups

11. What do you think the take home Methods messages were from the Langer and Rodin study?

• The take home message from the Langer and Rodin study was that quasi-experiments often assess the overall effectiveness of a treatment that has many components. The experiment evaluated the overall treatment "package," not the individual components. Follow up research then must be done to determine which components are critical for achieving the treatment effect. o Floor one floor 2 difference btw floors o Gather additional data ♣ Nurses ♣ Assistants o You can still ask a good scientific question, must be careful about gathering additional data to eliminate plausible alternatives o If they only gathered data from elderly folks o Trying to make strongest possible case from conclusion trying to draw o Difference was if they were given a plant, here's a plant im gonna take care of it ♣ One floor was control, one was going to take care of plant for you

2 What are the three main characteristics of a true experimental design, and how is control exerted?

• The three main characteristics of a true experimental design are that researchers manipulate an independent variable with treatment and comparison conditions, there is a high degree of control, and random assignment. Control is exerted especially through random assignment of conditions, so the two groups are equally balanced on secondary variables.

6. What do we mean by threats to internal validity and what are eight main types of these threats?

• Threats to internal validity are confounds that serve as plausible alternative explanations for a research finding. • Eight main types: o History o Maturation o Testing o Instrumentation o Regression o Subject attrition o Selection o Additive effects with selection. ♣ Big challenges whenever we don't get to randomly assign people because of different groups ♣ Selecting pre-existing groups ♣ Assumption is those groups are different to start with ♣ Maturation, history, groups might respond differently to external events ♣ One group maturing at different rate, respond to real world event different ways