ECON 440: Experimental Economics: Test 1, part 1 -Bachir Kassas
5. Other Aspects
. Random Lottery Incentive (RLI): · Usually used in within-subject designs or when subjects perform multiple tasks · Each subject completes a sequence of distinct tasks · Each task has a well-defined reward structure based on actions of subject and/or actions of other subjects and/or moves of nature · Subjects are made aware that when sequence of task is completed, one task is chosen at random and payment is made based on rewards in that task Advantages: · Provides a cheaper way of inducing incentives · Helps avoid wealth effects if payoffs from tasks influence subsequent decisions The Strategy Method: · Usually used in games of strategy (ultimatum game, trust game, etc.) · Contrary to the direct decision approach which asks each player to make one choice from the available set of possible actions, the strategy approach asks subjects to report an entire strategy table for every possible action played by the other player. · Example: in ultimatum games, instead of asking the responder to either accept or reject the extended offer, he is asked to write down his response to every possible offer the proposer can suggest. Advantages: · Provides a richer information set as it reveals how players would have behaved at information sets that might not arise in the actual course of play Repeated Games: Usually game theory experiments Subjects play repeatedly over a sequence of rounds Partner Design: The same group of subjects play together in every round Stranger Design: The groups are randomly shuffled each round Advantages: · Generate more data as you observe subject behavior at multiple rounds · Provides the opportunity for subjects to gain experience, which is important if we are interested in the behavior of experiences subjects and the learning process
Limitations of Experiments:
1. Control is never perfect • Weather, lab environment, exams, etc. • No real control over all other motives • Self selection: who takes part in experiment? -Knowledge?, Money? 2. Experiment compared to theory • Experiments are never general, just an example • Can we refute or validate a theory based on one experiment?-No 3. Experiments compared to field studies • Can all preferences be induced?-No (ie, time preferences) • Time preference
10. Types of Experiments
1. Laboratory Experiments: Usually short term and take place in a controlled environment (the lab). Relies on simple abstractions of the real world. • Advantages (internal validity): • The researcher has complete control over all conditions and variables in the setting. • Easy to eliminate confounding effects • Only Requires proper randomization of treatments to obtain reliable results. • Disadvantages (external validity): • Conducted in an artificial environment • Difficult to generalize the results to the more complicated real world. 2. Field Experiments: Can be short or long term and take place in the participant's natural environment • Advantages (external validity): • Conducted in a real world setting • Can be done without the participant's' knowledge that they are taking part in an experiment • Easier to generalize the results to the real world • Disadvantages (internal validity): • The researcher does not usually have complete control over all conditions and variables. • More vulnerable to confounding effects • Requires more complicated econometric models for data analysis
5. Main Uses of Experiments
1. Speaking to Theorists Testing theories under precisely controlled and/or measured conditions • Build experimental design based on theory • Implement conditions of theory • Compare predictions with experimental outcome Test several aspects of theory • Testing fundamental prediction of theory • Testing comparative static predictions of theory • Testing descriptive validity of assumptions about human behavior on which theory is based Explore the causes of failure of theory • Find out when theory fails and when it succeeds • Design proper control treatments that allow causal inference about why theory fails 2. Whispering in the Ears of Princes Influencing policymakers' decisions • Measuring the effect of new policies • Testing the impact of changes in existing policies Evaluate Policy Proposals • Does the reduction of entry barriers increase aggregate welfare? • Do emission permits allow efficient pollution control? • How should airport slots be allocated? The laboratory as a wind tunnel for new institutions • How does a privatized electricity industry with small numbers of suppliers and intermediate traders work? 3. Eliciting Preferences Measuring "home-grown" values of the individuals • How much should the government spend on avoiding traffic injuries? • How much should be spent on the conservation of nature? • Answer is important for tax policy, growth policy, etc. Helps us understand • How individuals make decisions • How individuals behave in interactive settings Assessing "fundamental" preferences: • Risk aversion, time preference, altruism/cooperation • What preferences underlie economic decision making • Useful for assessing policy impact, or predicting who will not respond • Natural disaster preparation, mortgage refinancing, subsidies for energy efficiency 4. Searching for Facts and Meaning Direct efforts of theorists • Discovering empirical regularities where theory has little to say • Formulating new theories to explain observed regularities • Using established regularities to assess and evaluate contradictory theories • In the presence of multiple equilibria (e.g., repeated games), experiments help us select most relevant ones 5. Exploring New Methods Testing features of new models and market structures • Study institutions in the lab before introducing them to the field • Compare different environments within the same institution (e.g., FPA with 5 bidders or 10 bidders) • Compare different institutions within the same environment (e.g., FPA vs. SPA) • Can do welfare comparisons even if theory about the effects of the institution are unavailable. 6. Pedagogical Using experiments in the classroom • Experimental demonstrations of economic propositions (market equilibrium) • Students experience economic concepts • Better understanding of economic phenomena by active learning • Thinking about economic questions differently
I. Subjects:
1. Who to Recruit? a. Students: Used in most Economics experiments (undergraduates or Masters students) • Advantages: • Readily accessible and convenient • Low opportunity cost (less costly) • Relatively steep learning curve for students • Disadvantages: • Narrowly defined population (not representative of general population) • Undermines external validity and generalizability of results Rule of thumb: Deciding against student subjects depends on the specific reasons why they might be considered less appropriate for the experiment on hand. b. Professionals: Used in some Economics experiments (consumers, traders, etc.) • Advantages: • More representative of general population. • Facilitates external validity. • Disadvantages: • They usually behave similar to what they usually do in real life (might disregard instructions). • Might undermine internal validity. Rule of thumb: If the objective of the experiment is different from what professionals usually do, they are bad subjects to recruit.
2. What is Experimental Economics?
A branch of Empirical Economics that focuses on individual behavior in a controlled environment, either in the lab or out in the field, in an attempt to create insights and predictions about real-world behavior
1. Purpose
A statement describing the general purpose of the experiment Should be specific enough to satisfy the curiosity of subjects Should be general enough not to reveal too much and induce experimenter demand effects! Experimenter Demand Effects: If the participants are able to form a precise idea of the kind of behavior the experimenter is looking for, this in itself might make such behavior more or less likely regardless of subjects' true preferences
Advantages:/Disadvantages:
Advantages: • Allows for estimation of interaction effects (interaction between treatments) Disadvantages: • Can be expensive depending on how many treatments we are interested in 3. Fractional Factorial Design: • Similar to the full factorial design except that only a subset of combinations are considered (not all combinations of treatments) • Example: Suppose we have two treatments and and we decide to place 30 subjects in a subset of combinations Advantages: • Can be significantly cheaper than full factorial • With treatments, full factorial requires while fractional factorial requires combinations Disadvantages: • Doesn't allow analysis of interaction effect 4. Block Design: • Subjects are divided into blocks according to some observable characteristic (eg. gender) • Randomization is performed within (not between) the blocks • Variable on which blocking is applied is called blocking factor and is a source of variability that is usually not of primary interest to experimenter Advantages: • Provides more accurate ATE estimates when subject pool is heterogeneous in certain observable variables Disadvantages: • Less suitable for large number of treatments 5. Within-Subject Design: • Each subject goes through all treatments • Can be thought of as a special case of blocking design (blocking factor is the subject) Advantages: • Impact of subject-specific effect is eliminated, which greatly improves the precision of ATE estimate (lower variance) Disadvantages: • Possibility of order effects (behavior of subjects depends on order in which treatments are experienced) 6. Crossover Design: • A within-subject design where the order of treatments is varied between subjects • Example: with 2 treatments A and B, half the subjects are assigned to the sequence AB and the other half to the sequence BA Advantages: • Solves the order effects problem in within-subjects designs Disadvantages: • Possible carryover effects (residual effect of treatment is previous period carrying over to current period). 7. ABA Design: • A within-subject design where subjects start with the control (A), then go through the treatment (B), and then are exposed to the control (A) again. Advantages: • Solves the carryover effects problem in within-subjects designs Disadvantages: • Cannot be used with treatments with irreversible effects
7. Threats to internal validity
All involve loss of control: • Incentives are inadequate • Subject is up to something • Experimenter demand effects • Selection (failure of randomization) • History (order effects) • Framing (language, examples inappropriate) • Instrumentation (is it a good measure? do you use the same one?) • Mortality (dropouts) • Interactions with selection.
HINTS! for test 1
Be able to: Draw Supply and dem curve from chart Calc consumer surplus/producer surplus No tax, maybe dwl
Chamberlin's Experiment - Pit Market
Chamberlin (1948) conducted a market experiment where he demonstrated that prices and quantities fail to converge to competitive equilibrium The aim of this experiment was to refute the competitive model Design • Buyers receive values and sellers receive costs • Buyers and sellers will meet in the center of the room and negotiate during a trading period • When a buyer and seller agree on a price, they will come together to the experimenter to report the price, which will be written on the blackboard • The buyer and seller will return to their seats and wait for the trading period to end Market efficiency Efficiency = how actual producer and consumer earnings relate to the maximum achievable earnings • Earnings = surplus • Consumer and producer surplus • Difference in efficiency between perfect competition and monopoly • Difference in efficiency with introduction of tax
Subjects - Ethics
Conducting research with human subjects entails some ethical concerns 1. Task: • No threat of moral, physical, or financial harm in Economics experiments • Minimal risk of psychological harm (subjects always have the choice not to begin or to end their participation at any time) 2. Reward: • Awarding grades based on participation can raise ethical questions if non-participants have no other way to earn the same bonus grades without participating in the experiment (usually assign homework to non-participants) • Awarding grades based on points earned in the experiment can be unethical if those grades are not a measure of student's learning objectives from the course 3. Information: • Deception is considered unethical in Economics experiments as it contaminates the subject pool and negatively impacts future research
Subjects - Human Subjects' Committee
Cost/Benefit analysis is usually required to evaluate a research idea before it is approved · Internal Review Board (IRB) is responsible for protecting the rights of the experimental subjects and ensuring that the experimental procedures comply with the standard acceptable protocol · Economics experiments are (mostly) harmless and are usually classified as minimal risk
4. Why Experimental Economics?
Data cannot be captured from naturally occurring conditions • Data is unavailable • Data contaminated by movement of several variables Greater control over experimental data • Allows implementation of ceteris paribus condition • Facilitates analysis (no sophisticated econometrics necessary) Replicability of experimental results
Vernon Smith's Experiment - Double Auctions
Design: • Subjects are randomly divided into two subgroups • Buyers • Sellers • Each buyer is given a card containing a number, known only to him, which represents his valuation for the good • Each seller is given a card containing a number, known only to him, which represents his cost (minimum acceptable price) • Payoff for buyer and seller • At any time, a buyer can raise his hand and say "buyer X bids ____" • At any time, a seller can raise his hand and say "seller Y offers ____" • Bids and offers are recorded on the board • At any time, a buyer can accept an outstanding offer • At any time, a seller can accept an outstanding bid • Buyers and sellers can trade only one unit of this fictitious good during a period • Subjects received economic incentives: induced value • Implementation of a game or economic institution • Rules for information and exchange Induced value: • Sellers received costs • Buyers received valuations Trading institution: • Double auction Main result: • Rapid convergence to competitive equilibrium across periods Note: Other trading institutions are possible!
Do's and Don'ts of Experimental Economics: Don'ts
Don'ts: Avoid loaded words! Loaded words in the instructions can (and often do) lead to experimenter demand effects which undermines the accuracy of the estimated treatment effects Avoid deception! Deceiving your subjects will definitely do more harm than good. If subjects doubt your credibility, they will start hedging against possible tricks (salience and dominance are compromised). Also, the subject pool will be contaminated, which affects future research Keep it short! Subjects' behavior will change as a result of fatigue and boredom. Try to keep your experimental sessions below 2 hours long to avoid this issue Avoid realism trap! Do not try to closely mimic reality in an experimental design as that will make your experiment more complex and impede your ability to disentangle causes and effect. Reality has infinite detail and can never be fully captured so keep it simple Don't reveal too much! Revealing too much about the purpose of the study or the goals of the experimenter can result in experimenter demand effects which will undermine the accuracy of the estimated treatment effects Avoid anchoring! Do not provide suggestive examples to your subjects on how they should respond to the tasks in the experiment. If you want to use examples for better illustration, you should provide more than one example using different extreme values to neutralize any anchoring effects Avoid income effects! Subjects general income levels, and their endowments and earnings in the experiment, might affect their behavior. If the effect of income is not of interest you should control this variable by equalizing endowments and expected earnings across your subjects Avoid confounding effects! If you change more than one factor at the same time, it might be impossible to disentangle individual effects Avoid losses or zero payoff options! This can create loss aversion scenarios where subjects change their behavior to avoid having to pay money (act extra conservative). Also, if a subject's payoff becomes zero he might start treating the task as hypothetical
Subjects - Individual Differences
Economic man populates our model § Simple agent, identical, maximizes happiness · Models where he lives do a very good job of describing and predicting aggregate behavior in a wide variety of settings § Markets, competitive environments, interactive environments · Subjects are individuals who are not uniform like experimental material in physics and engineering · Individual differences may exist and may be pronounced § Can be a problem (creates confounds in study) § Can be of interest What to do?: o If not interested in individual differences è dilute them § Randomize § Within subjects design § Collect data on individual characteristics to control them in analysis o If interested in individual differences § Treat subject characteristics as a factor in your design
2. Environment and Institution
Environment: • Agents (Number, type, motivation) • Commodities -- what do decisions get made over? • Endowments -- what do the decision-makers have at the outset? • Mechanism by which learning can occur (search opportunities, practice) Institution: • Decisions available to subjects • Rules about choices • Rules about communication • Connection between decisions and payoffs
11. Terminology
Factor: an element (variable) in an experimental design (i.e. level of information, experience, incentives, rules) Example: (FPA vs. SPA) and (low cost vs. high cost) Treatment: a unique environment or configuration of factors Example: (FPA low cost) or (SPA high cost) Session: A sequence of periods, games of other decision tasks involving the same group of subjects on the same day Cell: a set of sessions with the same experimental treatment conditions Experiment design: a specification of sessions in one or more cells to evaluate the propositions of interest
4. Induced Value Theory
How do we ensure that subjects reveal their true preferences in our experiment? Experimenter wants to control subjects' preferences (promote truthful reporting!) How? Neutralize "home-grown" preferences and induce new preferences • Subjects' actions must be driven by induced preferences Answer: Induce incentives for truthful reporting! Reward Medium: Money! Assumption: people care about money and some other motives • We can never be completely sure that subjects are revealing their true preferences, but inducing incentives can help increase our confidence that they are • This avoids unnecessary problems associated with responding to hypothetical scenarios (hypothetical bias) • Do you help others who are in need? • Give subject an opportunity to help! • Necessary to publish in economics!!! Three key characteristics are necessary in induced value theory: 1. Monotonicity: Subjects must prefer more of the reward medium (usually cash) to less and not become satiated 2. Salience: The reward received must depend on the subject's actions as defined by the institutional rules that he understands 3. Dominance: Changes in subjects' utility from the experiment come predominantly from the reward medium and other influences are negligible
Ways to Test Theory
Introspection: • Most typical in economics • Mathematical model (thought experiment) • Does the theory make sense? Observational field/happenstance data: • Low control • Requires econometrics Experiments: • Lab experiments • Field experiments Market Theory of Perfect Competition-Competitive Equilibrium Assumptions: • Agents are rational and selfish (profit maximizers) • Homogenous, well defined good is traded • Many producers and consumers • Producers are price takers • Producers and consumers have perfect knowledge of the market When these assumptions are satisfied, theory makes clear predictions about how the market will operate (equilibrium price and quantity, comparative statics) But, are these assumptions realistic? How well does theory translate into the real world?
9. Threats to external validity / Objections Against Experiments:
Lack of parallelism: • Are the relevant conditions in the experiment and the "real" world similar? • Does the experiment capture relevant conditions that prevail in reality? Examples: • Subject differences • Lab context Many reported failures of external validity are in fact failures of internal validity! Objections Against Experiments: 1. Experiments are unrealistic • Most economic models are unrealistic because they leave out aspects of reality • Experiments are models too! Answer: Simplicity of a model or experiment can be a virtue • Allows us to focus on effect of a few key variables Importance of realism depends on purpose of experiments • E.g., if purpose is to test theory or understand failure of theory then realism is not very important 2. Experiments are artificial • Biased subject pool (students) • low stakes • small number of participants • inexperienced subjects • anonymity This is not a fundamental objection. • Use other subjects • Increase the stake level • Increase the number of participants • Recruit experienced participants • Alter non-anonymity systematically Bottom line: criticisms can be tested in the lab
6. Classroom Experiments
Should you conduct experiments in your own scheduled classes or with your own students? Advantages: • Effortless recruitment and scheduling • Cheap • Can use grades instead of money as the reward medium Disadvantages: • Classroom relationship between subjects and experimenter can create internal and external validity issues • Subjects might change their responses to impress their instructor • Subjects might feel that their responses are somehow to their course or their overall grade
1. Main Branches of Economics:
Theoretical: • Develop theoretical models to understand economic behavior • Heavily based on mathematical abstractions of the real world • Generates predictions regarding equilibrium and comparative statics Empirical: • Relies on the collection and analysis of data (primary or secondary data) • Heavily based on regression models (structural form and reduced form) • Tries to uncover regularities in the data that can support or help refine theoretical models
Do's and Don'ts of Experimental Economics: Dos
This list summarizes the key points to watch out for when designing and running an experiment in economics Do's: Keep it simple! A simple design is a good design and avoids many of the ambiguities and confounds often encountered in experimental research Incentivize! Motivate subjects by paying them in cash depending on their actions and decisions. Maintain privacy! Maintain the privacy of subjects' actions and payoffs and of your own experimental goals. Maintaining privacy of subjects will encourage them to reveal their true preferences. Plan ahead! Must have enough data to do statistical tests. Decide what test you want to do and design experiment accordingly. Avoid too many treatments. Control non-institutional interactions! Experiments usually have break sessions and subjects can talk and disclose vital information in those sessions, which can affect their subsequent behavior. Carefully monitoring break sessions, or changing parameters after the break avoids this issue Control experience and learning! Subjects' behavior will change over time when they repeat a task as they learn and gain experience. If the effects of experience are not of interest, you should control it as a constant by using only experienced subjects Randomize! Random assignment of subjects to control and treatment groups helps avoid selection bias and generates more accurate estimates of ATE Don't forget the baseline! Must have something to compare to. Can be a replication of an earlier study or results from earlier study, but then must exactly match instructions. Control all controllable variables! Failure to control variables will make your data less informative than it could be Account for nuisance variables! If you suspect a nuisance variable is interacting with one of your focus variables, try controlling the nuisance variables as a treatment (two levels often suffice). When possible, it is better to control nuisance variables as constants to keep down complexity and cost Control lab environment! Controlling the lab environment will help you avoid unfavorable errors in data collection and collect clean data. For example, some subjects might unintentionally reveal private information while asking a question about the experiment. Lack of control over your lab environment will increase the probability of such mistakes
1. General Notes
Three ways to approach experimental design: 1. Begin with theory. Translate theory to lab. Test on its own domain E.g., Market experiment 2. Begin with a phenomenon. Design experiment to dissect this phenomenon. E.g., Giving in Dictator games 3. Begin with something you want to measure. Design experiment to measure it. E.g., Risk preferences
2. Purpose
To ensure you are testing what you think you are testing. • Primarily a design issue • Ensure there are no confounding factors • Generate independent observations for testing To ensure that your results can be replicated by others • Primarily a procedural issue • Standard lab procedures are followed and recorded
3. Examples and Quizzes
· Description should be elaborate enough to ensure everyone understands, but not too detailed to suggest a certain strategy or action · Avoid overly long instructions as subjects get bored and lose interest in fully understanding them · It is a good practice to use examples and quizzes when explaining the instructions to participants Quizzes: • Help test the subjects' level of understanding of the institution • If not carefully designed, they might give unintended cues about your underlying intentions to your subjects Examples: • Facilitate full comprehension of the institution and procedures • Might cause anchoring, where subjects would use the values presented in an example as an anchor and adjust their responses based on them • Best to present several examples, using extreme values on both ends (upper and lower) to avoid anchoring problems
3. Randomization
· Experimental research is mainly concerned with estimating treatment effects, which are best understood with the following framework: o Researcher is interested in effect of particular treatment § for treatment and for control § We are interested in the average treatment effect (ATE) o Let be outcome of subject given treatment status o The ATE is o We cannot observe and for all subjects • We can only observe for treated individual and for untreated individuals • What we can estimate is o Problem: if propensity to receive treatment is correlated with observed or unobserved subject characteristics then, è Our estimate of the ATE will be biased! o Solution: Randomization! o Randomization ensures and and generates an unbiased estimate of ATE There are several randomization techniques, each has advantages and disadvantages: 1. Completely Randomized Design: • Simplest design • Treatments are probabilistically assigned to subjects independent of any observed or unobserved characteristics • Example: With only a control and one treatment, each participant has 50% chance of being assigned to the control and 50% chance of being assigned to the treatment • Advantages: 1. Simple and easy to implement 2. Minimizes risk of correlation between treatment and subject characteristics • Disadvantages: 1. Sample sizes for each treatment are random 2. Variance of the outcome may be large 2. Full Factorial Design: • Assign a predetermined number of subjects to each combination of treatments • Subjects are still randomly assigned to each combination • Example: Suppose we have two treatments and and we decide to place 30 subjects in each combination
3. Sources of Data
· Happenstance data: a byproduct of ongoing uncontrolled processes · Experimental data: deliberately created under controlled conditions · Laboratory data: gathered in an artificial environment designed for scientific purposes · Field data: gathered in a naturally occurring environment All combinations are possible! (think of examples)
2. How Many to Recruit?
· Usually done on the basis of power analysis · Power is probability of correctly detecting a treatment effect. Or one minus probability of type II error () · Choose desired level of power (usually 80%) · Choose significance level of hypothesis test () · Perform power calculation to determine required sample size
8. External Validity:
• The issue of whether inference can be generalized from the lab to the field • Important for whispering in the ears of princes • More difficult to address (know which questions are suitable for experimental research) • Cannot convince skeptics using deductive arguments (since X, therefore Y). • Best to approach using inductive arguments (since the sun rises every morning, we are not sure it will rise tomorrow but can presume it) • Skeptics then have the burden to state what is different about the outside world that might change the results. • Use skepticism to promote constructive research, not engage in sterile arguments!
6. Internal Validity:
• The issue of whether the data permits causal inference • Concerns proper experimental controls and correct data analysis • Do the subjects understand what they are responding to? • Do they have incentives to reveal their true preferences? • Are their actions driven solely by the motivations the experimenter wants to investigate? • A properly designed experiment can mitigate internal validity issues
