What are the main advantages and disadvantages of experiments?
Unrepresentative subject pools (Disadvantages)
As noted above, subject pools may be unrepresentative of the populations of interest.
Experimenter bias (Disadvantages)
Experimenter bias, including expectancy effects and demand characteristics, can limit the relevance, generalizability, or accuracy of certain experimental results.
Precise measurement (Advantages)
Experimenters design and implement the desired measures and ensure that they are administered consistently.
Ability to explore the details of process (Advantages)
Experiments offer the opportunity to explore phenomena of interest in great detail. Complex relationships can be broken down and investigated in smaller units in order to see which part of the process results in the differences of interest. In addition, experiments allow particular relationships to be explored in the presence or absence of other variables, so that the conditions under which certain relationships hold can be examined as well.
Artificial environment (Disadvantages)
Many experimental settings are artificially sterile and unrepresentative of the environments in which subjects might normally per- form the behavior under study. There are at least two important aspects of this limitation. First, it might be impossible or unethical to create the desired situation within a laboratory. An experimenter could not study the effects of a life-threatening illness by causing such disease in a subject. Second, it may be very hard to simulate many phenomena of interest—an election, a war, an economic recession, and so on.
Experimental control (Advantages)
The experimenter has control over the recruitment, treatment, and measurement of subjects and variables.
Ability to derive causal inferences (Advantages)
The major advantage of laboratory experiments is in its ability to provide us with unambiguous evidence about causation. Because of the randomization of subjects and the control of the environment, experiments allow confidence regarding causal inferences about relationships among the variables of interest.
Relative economy (Advantages)
Although experiments may be more costly and time- consuming than some research methodologies, such as formal modeling, they are certainly more economical than conducting large surveys or field experiments. Students are a relatively inexpensive and reliable subject pool, and a large number of them can be run in a semester. Experiments embedded in larger surveys (Sniderman et al. 1991, Kuklinski et al. 1997) may provide a more representative sample but would also require a great deal more time and money to administer.
External validity (Disadvantages)
For political scientists, questions surrounding external validity pose the greatest concern with experimentation. What can experiments tell us about real-world political phenomena? Beyond the nature of the subject pool, this concern is at least twofold. First, in the laboratory it is difficult to replicate key conditions that operate on political actors in the real world. Subjects typically meet only for a short period and focus on a limited task. Even when money serves as a material incentive, subject engagement may be low. In the real world, actors have histories and shadows of the future with each other, they interact around many complex issues over long periods, and they have genuine strategic and material interests, goals, and incentives at stake. Can the results of a single-session experiment tell us anything about such a complicated world? Second, and related, many aspects of real-world complexity are difficult to simulate in the laboratory. Cultural norms, relationships of authority, and the multitask nature of the work itself might invalidate any results that emerge from an experiment that does not, or cannot, fully incorporate these features into the environment or manipulation (Walker 1976). In particular, subjects may behave one way in the relative freedom of an experiment, where there are no countervailing pressures acting on them, but quite another when acting within the constrained organizational or bureaucratic environments in which they work at their political jobs. Material and professional incentives can easily override more natural psychological or ethical concerns that might manifest themselves more readily in the unconstrained environment of the laboratory. Failure to mimic or incorporate these constraints into experiments, and difficulty in making these constraints realistic, might restrict the applicability of experimental results to the real political world. There are two important things to understand about external validity. First, external validity is only fully established through replication. Experiments testing the same model should be conducted on multiple populations using multiple methods in order to determine the external validity of any given experimental paradigm. Second, external validity is more closely related to the realism created within the experiment than to the external trappings of similarity to real-world settings, which is referred to as mundane realism. As long at the experimental situation engages the subject in an authentic way, experimental realism has been constructed; under these circumstances, mundane realism may be nice but is hardly required to establish causality. Moreover, even if the experiment closely approximates real-world conditions, if its subjects fail to engage in an experimentally realistic way, subsequent findings are useless.
