RS Week 5 Research Designs in Political Science: Quasi-Experiments, Natural Experiments, and Passive Observational Designs
ALWAYS ASK (in passive observational design)
HOW ARE THE GROUPS DIFFERENT ASIDE FROM THEIR DIFFERENCES ON X, AND HOW COULD THOSE DIFFERENCES ACCOUNT FOR THE RESULTS THAT ARE FOUND?
ALWAYS ASK (In QEWITHPT)
HOW WERE THE GROUPS DIFFERENT BEFORE X WAS INTRODUCED, AND HOW COULD THOSE DIFFERENCES ACCOUNT FOR THE RESULTS THAT ARE REPORTED?
problems in implementing the QE (pretest) design
SAME AS BEFORE - drop-out - "contamination" of the treatment and control groups through spillover effects - unknown variation in treatment
Examples: Political Science Natural Experiments
Some parts of East Germany were unable to receive West German television signals due to topography, so can compare East Germans who were exposed to West German television to those who did not (could not). "Nature" prevented some East Germans from watching, so we can compare those people to the other East Germans who could watch
"Selection Bias"
There is sometimes self-selection into the treatment group, and the kinds of things (Z) that lead individuals or units to select themselves into the treatment may also influence Y
Time series study
When you have one or only a few units with many time points
panel study
When you have only a few time points and many units
The better designs, however
are not always possible to implement, so we need to know how to maximize the power that even lesser designs can provide!
What causes the weaknesses in other designs
lack of pure randomization and the lack of the kind of researcher control over the treatment (X) as exists in the classical experiment
all of these designs have
many of the same weaknesses,
"Passive Observation" or "Ex Post Facto" Studies
observe a sample of units at one point in time, with some of them having higher values on X, some of them having lower values on X, and we compare the groups on some dependent variable(s).
"Selection - History" interaction
the treatment group differed from the control group, and this difference (not X) led to a reaction to something else aside from X that happened after the pre-test
"Selection - Maturation" interaction
the treatment group was already changing on Y at a faster pace than the control group
The QE design is
the workhorse of applied policy and evaluation research
How is passive observation different quasi- or natural experiment
there is no direct manipulation of X by the researcher or by nature in these kinds of designs. We simply observe different groups based on their values on X.
Advantages to Panel Studies
- Can analyze change in dependent variables directly - Can use previous values of Y as "pre-tests" to control for Zs - With enough waves, can take into account Y!X as well as X!Y - Can add more Zs into the analysis statistically with matching or other methods as well
As a general rule, we prefer designs with:
-pre-tests -more time points -more units -more researcher control over the introduction of the independent variable
Better Method to overcome potential Threats in Observation design
Add More Time Points or Periods of Observations
Add more time points
Add more observations on each unit so you can see how the dependent variable changes after an independent variable changes, and you can see more clearly the longer-term trends for groups that do and do not change on X over time
Example of Panel Studies
Example: Finkel (1985) on syllabus. Survey of US adults 1972-74-76 (a 3 "wave" panel). Ask about participation in elections and questions relating to political efficacy in all waves. Do people who participate in politics change in their efficacy over time more than people who don't participate? - Treatment group: People who participate at one point in time - Control group: People who don't - Dependent variable: Change in participation from one point to the next
Example of Quasi-Experiments without a Pre-Test
Example: Finkel's Civic Education and the Mobilization of Participation in New Democracies (Journal of Politics, 2002): estimate the effect of exposure to civic education workshops in Dominican Republic and South Africa by comparing participants with "matched" non-participants at one point in time, after the workshops had taken place
Examples of Observational Design
"Does the individual's financial conditions influence their vote?" - "Treatment" Group: Individuals whose financial situation got worse - "Control" Group: Individuals whose financial situation did not get worse - Single-shot observation: We conduct a representative survey of US adults, and we ask: Did people in our "Treatment Group" and our "Control Group" differ in their likelihood of voting for the incumbent party or opposition party candidates?
3 types of Selection Bias in a Quasi Experiment with pretest
"Selection - History" interaction "Selection - Maturation" interaction "Selection - Regression" effect
Biggest Problem with QE studies with a pretest
"Selection Bias"
Types of Time Series Studies
"Trend Analysis" "Intervention Analysis": "Multiple Time Series":
The Passive Observation design has exactly the same form and face exactly the same issues as another common kind of design in political science:
"post-test only" Quasi- Experiment
Strengths of the Single-Shot ("Cross- Sectional") Passive Observation Design
1. External validity can be *really* high if the study has a good representative sample. 2. this design is often all that the researcher can do when manipulation is not possible
What do you do when a Classical Experimental research design is not possible?
1. Quasi-Experiment 2. Natural Experiment 3. Observational Study
Potential Problems in Estimating Causal Effects with QE (pretest) Designs
1. Selection Bias 2. As with the classic experiment, the QE researcher may not know the true length of time it takes for X to cause Y 3. As with field experiments, there may be problems in implementing the QE design
Strengths of Quasi-Experiments with Pre-Tests
1. control for the baseline levels of Y for all groups - the starting points on the dependent variable 2. The reseacher directly observes changes in Y, and (usually) knows that X came before the changes in Y were observed 3. relatively high external validity, as they involve real world observations, without laboratory or other constraints imposed through classic experimental methods
Threats to Passive Observation
1. no baseline pre-test 2. The "treatment" and "control" groups may be different on a lot of Z variables that could confound the process 3. No time precedence
Examples of applied policy and evaluation research
Educational reforms
The Classic Quasi-Experiment
Experiments that have treatments, outcome measures, and experimental units, but do not use random assignment to create the comparisons from which treatment-caused change is inferred. Instead, the comparisons depend on non-equivalent groups that differ from each other in many ways other than the presence of the treatment whose effects are being tested
Examples of Quasi-Experiements with PreTest
Finkel Kenya Civic Education Study (2001-2002) - 1000 Kenyans interviewed as they entered one of 161 civic education workshops conducted in February-May 2002 in run-up to national elections; 1000 "control" individuals from same neighborhoods interviewed simultaneously. Individuals re- interviewed in December, just before elections take place. [Results suggest that civic education has positive effects on many democratic orientations, e.g. knowledge, tolerance, participation].
"Selection - Regression" effect
If people who are generally very low (or high) on Y are selected for treatment, their gains on Y will be disproportionately positive (or negative) compared to the control group even in the absence of a treatment effect
"Multiple Time Series":
Intervention analysis with a control group
"Trend Analysis": Examples:
Look at one case over a long period of time, correlate changes in Y with changes in X that take place over the long time horizon. Examples: Does presidential popularity depend on economic performance?
"Intervention Analysis": Examples:
Look at one case over a long time period, with a definite "intervention" or change in X that takes place sometime in that period. Did change in speed limit in country/ state lead to fewer traffic fatalities? - Example from Class Web Site: "How a Conservative-Led Australia Ended Mass Killings"
closest observational design to an experiment that we have in political science?
Panel Designs
The most common political science experiment design
Passive Observational design
• If you have two time points,then the "passive observational" panel design
has exactly the same form as the Classic Quasi- Experimental Design on slide 9. The only difference is whether the "treatment" X is observed passively or whether it is actively manipulated by the researcher.
The Natural Experiment
is a kind of passive design, but unlike most passive designs, it has some kind of active manipulation of X. The interesting feature is that the manipulation is done by "nature" and not by the researcher.
How to Overcome these Potential Threats? (QEWITHPT)
• "Matching" of the treatment and control groups on as many Z variables as possible before the treatment is introduced, so that the groups are as "balanced" as possible in a non-randomized design. • Use the "oversubscription" or "phased roll-out" methods of implementation of treatment to ensure control groups that are as similar as possible to the treatment groups on both observables and unobservables. • Statistical controls for Zs in the data analysis phase. See if X still leads to changes in Y once the Zs are controlled.
Quasi-Experiments without a Pre-Test:
• A less powerful version of the QE has the same general features but no pre-test. That is, some stimulus (independent variable) is administered to a non- randomly-assigned treatment group and the levels of dependent variable are compared to a non-randomly- assigned control group.
Characteristics of the Quasi Experiment
• Non-random or voluntary assignment to treatment or control group • Less control over the form and the level of X, i.e., the treatment • Pre-test to measure change • As in classic experiment, the "difference in difference" is the estimation of the causal effect of X
How to Overcome these Potential Threats in Observation design
• Statistical controls for Zs in the data analysis phase. • But this cannot control for confounding due to unobservables, which, by definition, the researcher has not measured or included in the study
Problems with Causal Inference in Quasi-Experiments without a Pre-Test
• The lack of randomization and lack of pre-test are crucial to control confounding factors and starting points • Sometimes we use the measurement process to find out how much X a person/unit was exposed to, and since this takes place afterwards, there may be more inaccuracies in our recording of this information than in the QE design with pre-test
Ruling out selection bias
• The presence of a pre-test is very helpful in ruling out some kinds of selection bias. • But there are still problems with selection bias, all of them rooted in the non-random assignment to treatment and control
Characteristics of Natural Experiment
• There is less control (obviously) about the level and form of X than in the true experiment • Depending on whether observations are gathered before nature manipulated X, we have either a classical Pre-Post set-up, or a "post-test only" set up • The more "random" the natural manipulation, the more the design resembles a true experiment. Scholars would like to be able to claim that the manipulation by nature was "as if" random; to the extent that this is plausible, it gives the design more power
