Experimental designs and internal/external validity, March 29

Types of Experimental Designs

-Pre-experimental:limited in controlling for threats to internal validity -quasi-experimental: nonequivalent control group design -Experimental: randomized control group design

Primary ways to compensate for lack of randomization

1. Utilize a non-equivalent control group design 2. substitute "statistical controls" for the absence of physical control of the experimental situation and the possibility of selection bias 3. using multiple pre-tests and before the intervention so that you can compare the post test to a number of data points to see if there were trends existing prior to intervention. Simple Interrupted Time Series Design O1 O2 O3 O4 X O5 O6 O7 O8 Multiple Interrupted Time Series Design N1 O1 O2 O3 O4 X O5 O6 O7 O8 N2 O1 O2 O3 O4 X O5 O6 O7 O8 4. Switching replications (similar to the wait list randomized control design) N O1 Xa O2 N O1 O2 Xa O3

Quasi experimental

Cannot randomly assign participants to treatment but will compare the two groups at baseline (DV and other relevant factors) to ascertain similarities or differences Nonequivalent control group design N O1 X O2 N O1 O2

Experimental

Classic experimental Design: RO1 X O2 RO1 O2 Process: -Random assignment to control and experimental groups -measure DV for both groups -administer stimulus to experimental group -measure DV again for both groups -statistically compare the two groups' scores

Experimental

Classic: R O1 X O2 R O1 O2 Process: 1.random assignment to control and expertimental groups 2.measure DV for both groups 3. Administer stimulus (program) to experimental group 4. measure DV again for both groups 5. statistically compare the two groups' scores

External validity

Generalizability of Findings: across different populations across different settings across time

Diagramming a study

O="observation" X=intervention; treatment, stimulus R=random assignment N=non-equivalent groups

Statistical regression

Occurs when subjects placed in groups based on extremely high or low scores, people regress toward the mean

Pre Experimental

Post-test only: X O One group pre-test post-test design O1 X O2 Pre-experimental X O O

Random Assignment vs. Random Selection

Random selection is concerned with external validity--> the purpose is to generate a sample that represents the larger population Random assignment is concerned with internal validity --> the purpose is to use the process of randomization to divide the sample into 2 or more probabilistically similar groups (usually the sample selection is based on non-probability sampling strategies, ie convenience.)

Variations of experimental treatment studies

Treatment as Usual R O1 Xa O2 R O1 Xb O2 Xa=new treatment; Xb= standard treatment Waitlist Control R O1 Xa O2 R O1 O2 Xa O3

Selection-internal validity

concern that the kinds of people selected for one experimental condition differ from the people selected for the other condition

Internal Validity

confidence that the results of an experiment are directly linked to the IV(Stimulus/treatment) with a particular group at a particular time//Did the IV really cause the change or was it something else?

Internal Validity

confidence that the results of an experiment are directly linked to the IV(stimulus/treatment) with a particular group at a particular time

hawthorne effect-internal validity

mere presence of others watching your performance causes a change in your performance

Differential droupout/attrition

people drop out of the experimental and control groups at different rates and for different reasons

Characteristics of Quasi-experimental research

primarily distinguished from experimental designs by lack of random assignment of participants to treatment various methods are used to compensate for lack of randomization and to identify the extent to which change coincides with the intervention and not with other factors (history, maturation, testing)

Experiment

procedure used in the service of testing a theory ( a hypothesis derived from theory) in order to support them or disprove them -an experiment can be repeated and has a well defined set of both procedures and outcomes (replicable)

Matching plus randomization

randomization does not guarantee that the experimental and control groups are comparable The best way to ensure comparability is to first match pairs of participants on important variables (gender, level of ed., motivation) and then randomly assign them to the experimental and control groups

History-internal validity

refers to external changes or events that occur that affect large numers of sample members and which could account for pretest-posttest changes

Maturation-internal validity

refers to processes within the respondent operating as a function of the passage of time; refers to internal changes, either physical or psychological

testing-internal validity

refers to the effect of the exposure to a pretest that may cause people to change regardless of whether receive the treatment or not

Instrumentation-internal validity

refers to way variables are measured; measurement may change in a systematic way due to changes in instrument, observers, or scorers which may produce changes in outcomes during course of study.

Threats to External Validity

sample selection: did the people volunteer, do they have special characteristics sample attrition: what would be different about the characteristics of the people who remained vs. those who dropped out location of study: single site vs. multi site time of study: what were the social conditions when study was conducted? recession, economic factors, etc?

Research Design

structured and logical system for carrying out the study based on the purpose of the study 1. logical plan for answering the research question 2. method for collecting data Two Basic Types: correlational/descriptive experimental/causal

Fidelity of the Intervention

the extent to which the experimental condition is implemented according to the guidelines, especially over time

Diffusion of treatment/Contamination of control condition

the treatment provided to the experimental group spreads or diffuses into the control group/ contamination of control condition: recipients of experimental and control groups interact with one another --> spillover of improvements

Experimental Mortality/Attrition

when subjects drop out of the study, those left in the study may be different in some way

Resentful Demoralizations

when those not involved in the experimental condition as staff or recipients decrease in motivation and performance--> exaggerated large group differences at the end of the study

Compensatory rivalry

when those not involved in the experimental condition as staff or recipients increase their efforts to 'improve' to challenge the perceived inequity --> exaggerated small group differences at the end of the study