Lesson 9: Experimental Research
Extraneous
factors one does not control but has to live with, such as the weather
Matching
involves assigning subjects in a way that a particular characteristic is the same in each group • This is a technique for controlling systematic error
Test Marketing
is the most common field experiment • Projecting results - external validity is a key consideration in designing a test-market
Experimenter bias
occurs when an experimenter's presence, actions, or comments influence the subjects' behavior or sway the subjects to slant their answers in cooperation. - Source of Systematic Error.
Categorical variables
take on a value to represent some classifiable or qualitative aspect •Ex: advertising message (message "A" compared to message "B"; discrete category
treatment variable
the independent variable manipulated during an experiment to measure its effect on the dependent variable
independent variable
variables one controls directly such as price, packaging, product features, etc.
dependent variable
variables one does not directly control such as sales or customer satisfaction
Experimental Characteristics
• In an experiment, Experimental Variables are the independent variables • The researcher creates the variables rather than just measure them • Experimental Effects exist if you can measure a meaningful difference between the Experiment Group and the Control Group
covariate
A continuous variable included in the statistical analysis as a way of statistically controlling for variance due to that variable.
placebo
A false experimental treatment disguising the fact that no real treatment is administered.
maturation effect
A function of time and the naturally occurring events that coincide with growth and experience.
experimental group
A group of subjects to whom an experimental treatment is administered.
control group
A group of subjects to whom no experimental treatment is administered.
testing effects
A nuisance effect occurring when the initial measurement or test alerts or primes subjects in a way that affects their response to the experimental treatments.
instrumentation effect
A nuisance that occurs when a change in the wording of questions, a change in interviewers, or a change in other procedures causes a change in the dependent variable.
mannipulation check
A validity test of an experimental manipulation to make sure that the manipulation does produce differences in the independent variable.
confound
An experimental confound means that there is an alternative explanation beyond the experimental variables for any observed differences in the dependent variable
Counterbalancing
Attempts to eliminate the confounding effects of order of presentation by requiring that one fourth of the subjects be exposed to treatment A first, one fourth to treatment B first, one fourth to treatment C first, and finally one fourth to treatment D first.
blocking variables
Categorical variables included in the statistical analysis of experimental data as a way of statistically controlling or accounting for variance due to that variable.
Manipulation of the Independent Variable
Experimental independent variables represent hypothesized causal influences • I.e. if I change this condition (cause), what happens to the outcome (effect)
systematic or nonsampling error
Occurs if the sampling units in an experimental cell are somehow different than the units in another cell, and this difference affects the dependent variable. Think about the impact on the Snack Pack research experiment if one of the test units (cells) had much larger families than another cell. Your results would likely be skewed. So, you have to be careful with how you choose your subjects and do your best not to introduce unintended differences.
demand effect
Occurs when demand characteristics actually affect the dependent variable. • Example: during the introduction of the experiment, the interviewer explains the process differently to one group than to another. (experimenter bias)
history effect
Occurs when some change other than the experimental treatment occurs during the course of an experiment that affects the dependent variable.
test-market sabotage
Intentional attempts to disrupt the results of a test-market being conducted by another firm.
within-subjects design
Involves repeated measures because with each treatment the same subject is measured.
external validity
Is the accuracy with which experimental results can be generalized beyond the experimental subjects.
constancy of conditions
Means that subjects in all experimental groups are exposed to identical conditions except for the differing experimental treatments.
mortality effect (sample attrition)
Occurs when some subjects withdraw from the experiment before it is completed.
experimental condition
One of the possible levels of an experimental variable manipulation.
cohort effect
Refers to a change in the dependent variable that occurs because members of one experimental group experienced different historical situations than members of other experimental groups.
cell
Refers to a specific treatment combination associated with an experimental group.
field experiments
Research projects involving experimental manipulations that are implemented in a natural environment.
Key Variables
-Independent -Dependent -Treatment -Extraneous
Test Marketing Advantages & Disadvantages
1. Advantages of test marketing • Real-world setting • Researchers can easily communicate results to management 2. Disadvantages of test marketing • Cost: expensive • Time: Test-markets cannot be put together overnight. Simply planning a test-market usually takes months. Actually implementing one takes much longer. • Loss of secrecy: one drawback to actual field experimentation is that the marketplace is a public forum. Therefore, secrets no longer exist.
Test Marketing: Three broad uses:
1. Forecasting new product success • A product is marketed on a small scale under actual market conditions 2. Testing the marketing mix • A manager can study any element of the marketing mix with a test-market 3. Identifying product weaknesses • Test-market experimentation also allows identification of previously undetected product or marketing plan weaknesses
Internal Vs. External Validity
1. Internal validity exists to the extent that an experimental variable is truly responsible for any variance in the dependent variable • Ex: Manipulation check is a validity test of an experimental manipulation to make sure that the manipulation does produce differences in the independent variable 2. External Validity is the accuracy with which experimental results can be generalized beyond the experimental subjects • Ex: Convenient samples like students are often used as subjects, but can you project findings from that group to a larger population?
Popular U.S. Test Market cities
1. Nashville, TN 2. Cincinnati, OH 3. Indianapolis, IN 4. Charleston, SC 5.J acksonville, FL 6. Greenville, SC 7. Oklahoma, OK 8. Phoenix, AZ 9. Albuquerque, NM 10. Winston, NC The goal in picking a test market city is to be able to project the results to the entire country. As a result, popular test markets most closely match the US population on demographic factors like income, ethnicity, population density, education, etc.
Reducing Demand Characteristics
1. Use an experimental disguise • Don't tell the subjects exactly what you are studying and try to disguise the real purpose of the experiment • Ex: Use a Placebo - a false experimental treatment disguising the fact that no real treatment is administered to that test cell 2. Use a "blind" experimental administrator • The researcher administrator doesn't know what is being studied, so will be less likely to unconsciously bias the experiment 3. Administer only one experimental condition per subject 4. Avoid using subjects paid on task performance 5. Avoid using professional subjects - Use an experimental disguise. - Isolate experimental subjects. - Use a "blind" experimental administrator. - Administer only one experimental treatment level to each subject.
tachistoscope
Device that controls the amount of time a subject is exposed to a visual image.
interaction effect
Differences in dependent variable means due to a specific combination of independent variables.
between-subjects design
Each subject receives only one treatment combination.
internal validity
Exists to the extent that an experimental variable is truly responsible for any variance in the dependent variable.
Control over Extraneous Variables
Experimental confounds mean that an alternative explanation exists beyond the experimental variables • Once a potential confound is identified, the validity of the experiment may be called into question
demand characteristic
Experimental design element or procedure that unintentionally provides subjects with hints about the research hypothesis.
Basic Issues in Experimental Design
Experimental designs involve four important elements: 1. manipulation of the independent variable 2. selection and measurement of the dependent variable 3. selection and assignment of experimental subjects 4. control over extraneous variables
experimental treatment
The term referring to the way an experimental variable is manipulated.
Summary of Experimental Characteristics
Experiments differ from ordinary survey research. The differences can be understood by identifying characteristics of experiments. These characteristics include the following: - Experiments use subjects instead of respondents. - Experimental variables become the key independent variables. The researcher creates the experimental variables rather than simply measuring them. Measured independent variables are called blocking variables or covariates in experiments. - Experimental effects are determined by differences between groups formed in the experiment. Main effects are differences in the means based on a single variable. Interaction effects are dif- ferences in means based on combinations of two or more variables.
repeated measures
Experiments in which an individual subject is exposed to more than one level of an experimental treatment. • are experiments that expose an individual subject to more than one level of an experimental treatment • Ex: Patients given medication and then given exercise as a treatment
placebo effect
The effect in a dependent variable associated with the psychological impact that goes along with knowledge of some treatment being administered.
main effect
The experimental difference in dependent variable means between the different levels of any single experimental variable.
Experiments are widely used in causal research designs
The experimenter manipulates one or more independent variables and holds constant all other possible independent variables while observing effects on dependent variables - A researcher can control variables in an experiment to a degree not possible in a survey
randomization
The random assignment of subject and treatments to groups; it is one device for equally distributing the effects of extraneous variables to all conditions.
Laboratory Experiment
The researcher has more complete control over the research setting and extraneous variables. - the researcher maximizes control over the research setting and extraneous variables
subjects
The sampling units for an experiment, usually human respondents who provide measures based on the experimental manipulation.
SELECTION
The selection effect is a sample bias that results from differential selection of respondents for the comparison groups, or sample selection error, discussed earlier.
test units
The subjects or entities whose responses to the experimental treatment are measured or observed - Sample selection and random sampling errors
Continuous variables
can take on any value •Ex: price; continuous Researcher should decide before the experiment which kind of independent variables are most relevant to the study
Selection/Measurement of Dependent Variable
• Selecting dependent variables is crucial in experimental design • Unless the dependent variables are relevant and truly represent an outcome of interest, the experiment will not be useful • Choosing the right dependent variable is part of the problem definition process • Ex: Crystal Pepsi - high trial, but low repeat The Crystal Pepsi case is a good example of the importance of choosing the right dependent variable. If you had selected initial sales, you would have predicted that it would be very successful. However, you'd be wrong. Trial wasn't the issue. The problem was repeat purchase. Plenty of people tried it for the novelty, but not enough thought it worth buying again. Therefore, you have to be careful that you are considering the whole picture.
Identifying extraneous variables
• Was the demographic makeup of the groups the same? • How did the control group fill the time consumed by the experimental group in being exposed to the experimental treatment? • Were the two groups of the same general achievement profiles?