Experimental designs and internal/external validity, March 29
Types of Experimental Designs
-Pre-experimental:limited in controlling for threats to internal validity -quasi-experimental: nonequivalent control group design -Experimental: randomized control group design
Primary ways to compensate for lack of randomization
1. Utilize a non-equivalent control group design 2. substitute "statistical controls" for the absence of physical control of the experimental situation and the possibility of selection bias 3. using multiple pre-tests and before the intervention so that you can compare the post test to a number of data points to see if there were trends existing prior to intervention. Simple Interrupted Time Series Design O1 O2 O3 O4 X O5 O6 O7 O8 Multiple Interrupted Time Series Design N1 O1 O2 O3 O4 X O5 O6 O7 O8 N2 O1 O2 O3 O4 X O5 O6 O7 O8 4. Switching replications (similar to the wait list randomized control design) N O1 Xa O2 N O1 O2 Xa O3
Quasi experimental
Cannot randomly assign participants to treatment but will compare the two groups at baseline (DV and other relevant factors) to ascertain similarities or differences Nonequivalent control group design N O1 X O2 N O1 O2
Experimental
Classic experimental Design: RO1 X O2 RO1 O2 Process: -Random assignment to control and experimental groups -measure DV for both groups -administer stimulus to experimental group -measure DV again for both groups -statistically compare the two groups' scores
Experimental
Classic: R O1 X O2 R O1 O2 Process: 1.random assignment to control and expertimental groups 2.measure DV for both groups 3. Administer stimulus (program) to experimental group 4. measure DV again for both groups 5. statistically compare the two groups' scores
External validity
Generalizability of Findings: across different populations across different settings across time
Diagramming a study
O="observation" X=intervention; treatment, stimulus R=random assignment N=non-equivalent groups
Statistical regression
Occurs when subjects placed in groups based on extremely high or low scores, people regress toward the mean
Pre Experimental
Post-test only: X O One group pre-test post-test design O1 X O2 Pre-experimental X O O
Random Assignment vs. Random Selection
Random selection is concerned with external validity--> the purpose is to generate a sample that represents the larger population Random assignment is concerned with internal validity --> the purpose is to use the process of randomization to divide the sample into 2 or more probabilistically similar groups (usually the sample selection is based on non-probability sampling strategies, ie convenience.)
Variations of experimental treatment studies
Treatment as Usual R O1 Xa O2 R O1 Xb O2 Xa=new treatment; Xb= standard treatment Waitlist Control R O1 Xa O2 R O1 O2 Xa O3
Selection-internal validity
concern that the kinds of people selected for one experimental condition differ from the people selected for the other condition
Internal Validity
confidence that the results of an experiment are directly linked to the IV(Stimulus/treatment) with a particular group at a particular time//Did the IV really cause the change or was it something else?
Internal Validity
confidence that the results of an experiment are directly linked to the IV(stimulus/treatment) with a particular group at a particular time
hawthorne effect-internal validity
mere presence of others watching your performance causes a change in your performance
Differential droupout/attrition
people drop out of the experimental and control groups at different rates and for different reasons
Characteristics of Quasi-experimental research
primarily distinguished from experimental designs by lack of random assignment of participants to treatment various methods are used to compensate for lack of randomization and to identify the extent to which change coincides with the intervention and not with other factors (history, maturation, testing)
Experiment
procedure used in the service of testing a theory ( a hypothesis derived from theory) in order to support them or disprove them -an experiment can be repeated and has a well defined set of both procedures and outcomes (replicable)
Matching plus randomization
randomization does not guarantee that the experimental and control groups are comparable The best way to ensure comparability is to first match pairs of participants on important variables (gender, level of ed., motivation) and then randomly assign them to the experimental and control groups
History-internal validity
refers to external changes or events that occur that affect large numers of sample members and which could account for pretest-posttest changes
Maturation-internal validity
refers to processes within the respondent operating as a function of the passage of time; refers to internal changes, either physical or psychological
testing-internal validity
refers to the effect of the exposure to a pretest that may cause people to change regardless of whether receive the treatment or not
Instrumentation-internal validity
refers to way variables are measured; measurement may change in a systematic way due to changes in instrument, observers, or scorers which may produce changes in outcomes during course of study.
Threats to External Validity
sample selection: did the people volunteer, do they have special characteristics sample attrition: what would be different about the characteristics of the people who remained vs. those who dropped out location of study: single site vs. multi site time of study: what were the social conditions when study was conducted? recession, economic factors, etc?
Research Design
structured and logical system for carrying out the study based on the purpose of the study 1. logical plan for answering the research question 2. method for collecting data Two Basic Types: correlational/descriptive experimental/causal
Fidelity of the Intervention
the extent to which the experimental condition is implemented according to the guidelines, especially over time
Diffusion of treatment/Contamination of control condition
the treatment provided to the experimental group spreads or diffuses into the control group/ contamination of control condition: recipients of experimental and control groups interact with one another --> spillover of improvements
Experimental Mortality/Attrition
when subjects drop out of the study, those left in the study may be different in some way
Resentful Demoralizations
when those not involved in the experimental condition as staff or recipients decrease in motivation and performance--> exaggerated large group differences at the end of the study
Compensatory rivalry
when those not involved in the experimental condition as staff or recipients increase their efforts to 'improve' to challenge the perceived inequity --> exaggerated small group differences at the end of the study