Chapter 8: Experimental Research Review

What should Dr. Wallace be concerned with?

Salience *It is clear that, due to a technology error, the salience of the manipulation--that the participants were aware of the independent variable--was very low or even nonexistent."

Double-blind experiments are an important defense against ____.

Experimenter effects *Experimenter effects occur when a researcher subtly or unconsciously affects the performance of a study participant. Both verbal and nonverbal cues, such as nodding or an encouraging tone of voice, may nudge the participant into behaving in such a way as to confirm the study's hypothesis.*

Survey experiments are increasingly becoming ____, with the Time-Sharing Experiments for the Social Sciences (TESS) program as a primary example.

Web-based *The Time-Sharing Experiments for the Social Sciences (TESS) program allows social science researchers to run experiments online using a nationally representative panel of participants, providing "researchers the opportunity to capture the internal validity of experiments while also realizing the benefits of working with a large, diverse population of research participants."*

Which type of experiment is an audit study?

Field experiment *Audit studies are a type of field experiment that presents scenarios where an individual makes a choice among matched pairs (or more) who differ on a single (or sometimes multiple) characteristics.*

Which type of experiment generally has the highest levels of internal validity?

Laboratory experiment *In general, experiments where researchers can cleanly manipulate the independent variable while holding other factors constant and can randomly assign participants to conditions have the highest internal validity. That is, they will produce data that can most convincingly evaluate whether changes in the independent variable cause changes in the dependent variable.*

If a social scientist has limited funds and time but needs to make use of a manipulation to best address causality, which type of experiment would work best?

Laboratory experiment *Laboratory experiments are often cheap and quick because they can be done with undergraduate participants.*

Which type of experiment does not allow for random assignment?

Natural experiment *A disadvantage of a natural experiment is that the researcher cannot randomly assign participants to the conditions. For example, a researcher studying "blind" orchestra auditions obviously could not assign some women and men to audition in the earlier historical period when their gender was known and others to the later period when screens were introduced.*

Population-based survey experiments differ from traditional surveys. In both, participants may read a description of a scenario and then answer questions about how they would react to the given situation. However, in a survey experiment, researchers ____.

Vary the independent variable by changing key features of the scenario *In a survey experiment, participants often read a description of a scenario and then answer questions about how they would react to the given situation. What makes a survey experiment an "experiment" is that researchers vary the independent variable by changing key features of the scenario, thus creating the experimental conditions. Participants are then randomly assigned to read one version of the scenario and answer survey questions designed to measure the dependent variable.*

Which type of design did Devah Pager use in her famous study on the effects of race and criminal record among employers?

Within-subjects design *Participants in a within-subject design receive all levels of the independent variable. In Pager's experiment, one tester with a criminal record and one without both applied for the same job, so each employer received both levels of the criminal status independent variable (criminal and noncriminal).*

Match each type of experiment with the example that best fits. 1) Natural Experiment 2) Laboratory Experiment 3) Field Experiment 4) Population-Based Survey Experiment A) A sociologist examines how the organization of public schools into charter schools destroyed by Hurricane Katrina affected children's learning compared to those whose schools reopened as regular public schools B) A sociologist emails real estate agents asking if property is available but alter the name each email comes from C) A sociologist conducts a telephone survey in which she presents participants with one of two different types of family scenarios and asks whether the respondents think the family is deserving of government assistance D) A sociologist invites undergraduates to evaluate résumés that list different types of educational credentials but are otherwise the same

1) A 2) D 3) B 4) C

Match each type of measure with the example that best fits. 1) Physiological Measure 2) Behavioral Measure 3) Attitudinal Measure A) How often a participant responds to email inquiries B) The level at which a participant agrees with the statement "Racial discrimination is a serious problem in this country today" C) The amount of sweat produced during an interaction with police

1) C 2) A 3) B

How many conditions must be met to establish causality?

3 *To establish that one variable causes changes in the other variable, three conditions must be met: (1) the two variables must be correlated, or co-vary, meaning an increase (or decrease) in one variable is associated with an increase (or decrease) in the other; (2) the cause (independent variable) must precede the effect (the dependent variable); and (3) the relationship between the independent variable and dependent variable must not be spurious, or caused by some other factor.*

Dr. Thomas designs a laboratory experiment to examine how people respond to criticism differently depending on if it comes from a man or a women. She tells participants that they are communicating electronically with another student about arts participation programs. However, the "other student" is really a computer script that steers the conversation toward personal criticism. What has Dr. Thomas used to avoid speculation and suspicion about the true purpose of the study?

A cover story *To keep participants engaged without revealing the study's true purpose, researchers often create a cover story, which is a false reason for participation.*

Which of the following statements about experiments is true?

Because of their artificiality, experiments are well suited for testing abstract theories about how the social world works *Experiments uniquely solve the problem of spuriousness. By experimentally holding other factors constant and varying only the independent variable, we are assured that the relationship between the independent variable and the dependent variable is not spurious. Experiments are the best research method to establish causality. Experiments can uncover mechanisms that produce an outcome, establishing not only if employers discriminate against mothers, but also why. Experiments can be used to evaluate abstract theories about the workings of the social world.*

Dr. Martin designs a laboratory experiment to examine how people respond to unexpected rude interactions with strangers. She tells participants that they should come to the lab to fill out a survey. However, she has set it up so that one of her assistants will intentionally bump into participants as they come to the lab and say "Watch yourself!". What has Dr. Martin used to help manipulate the independent variable?

Confederates *Sometimes researchers rely on confederates to help manipulate the independent variable. Confederates are individuals who interact with participants in the experiment while pretending to be participants themselves. In reality, they are part of the research team, trained to enact certain roles or behaviors.*

Because experiments can use deception and cover stories, it is important for researcher to ____ participants and tell them ____.

Debrief; The true purpose and goals of the study *Debriefing should be done with great sensitivity, as participants may understandably be upset upon learning that they have been deceived. Experimenters should look for signs that a participant is upset and have a set of procedures for dealing with upset participants, including where to refer them for further help.*

Experimental researchers often attempt to increase the external validity of their study by (1) replicating the experiment across different samples of participants, (2) replicating the study with a different kind of experiment, or (3) ____.

Evaluating the hypothesis with a nonexperimental research method *A final way to increase external validity is to use different types of research methods to study the same general research question, a practice called triangulation. When many different types of research methods find similar results, our confidence in the generalizability of the conclusions increases.*

Compared with other research methods, ____ most effectively evaluate whether a change in an independent variable causes a change in the dependent variable.

Experiments *Experiments are the best method for establishing cause-effect relationships, or causality.*

Dr. Butler conducts an audit study to examine how politicians respond to emails from constituents. He sends four emails to each politician on his list. The emails vary the name of the sender to suggest race (white or black) and gender (male or female). Which type of design has Dr. Butler included in his audit study?

Factorial design *A factorial design experiment is an experiment that has two or more independent variables. In this case, Dr. Butler has two independent variables (race and gender). Additionally, his study is not a between-subjects design because all politicians receive all four combinations (race x gender) of independent variables.*

Which of the following statements about field experiments is true?

Field experiments are better at assessing whether an outcome occurs than at evaluating the process that produces the outcome *Field experiments frequently rely on nonrandom samples. Another disadvantage of field experiments is that researchers have less control over other features of the setting. However, field experiments occur in more realistic and consequential settings, thus leading to higher levels of external validity than laboratory experiments. Finally, field experiments are better at assessing whether an outcome occurs than at evaluating the process that produces the outcome. So, for example, while field experiments are excellent for assessing whether a group is discriminated against, they are less helpful for figuring out why the discrimination occurs.*

Experiments share three key features: (1) manipulation of the ____ variable, (2) random assignment of participants to ____ conditions, and (3) experimental control of other factors that could influence the outcome of the experiment.

Independent; Experimental and control *An experiment is a research method where the researcher manipulates one or more independent variables to determine the effect(s) on a dependent variable. Experiments share three key features: (1) manipulation of the independent variable, (2) random assignment of participants to experimental and control conditions, and (3) experimental control of other factors that could influence the outcome of the experiment.*

In the United States, a researcher would have to justify the use of deception, lack of consent, and other ethical issues to a(n) ____ board.

Institutional review *IRBs are committees located at institutions where research is conducted, such as universities and colleges, hospitals, and survey research firms. Any institution that receives research money from the federal government, through organizations such as the National Science Foundation (NSF) or National Institutes of Health (NIH), must have an institutional review board. This board is responsible for a thorough review of all research conducted at the institution, not just federally funded research. Researchers must receive approval from these boards before they begin their research.*

Which of the following is jeopardized when participants do not notice an experimental manipulation?

Internal validity *In this case, the conclusions suggest that skin tone does not matter. The lack of salience in the independent variable among participants jeopardizes that conclusion and the internal validity of the study because the signal of the independent variable was weak. If we could establish that the differences in skin tone were conveyed clearly, internal validity should be very high.*

What is the purpose of random assignment in experiments?

It ensures that the only difference between the experimental group and the control group is the independent variable *Random assignment ensures that individual characteristics such as personality traits, biological traits, family background, and attitudes are randomly distributed across conditions.*

Goldin and Rouse's (2000) study of gender discrimination in the hiring of orchestra musicians is an example of which type of experiment?

Natural experiment *Starting in the 1970s and 1980s, orchestras gradually begin switching to blind auditions. Previously, male and female musicians auditioned in front of a team of evaluators. The introduction of the screen created a natural manipulation of the independent variable, which created two conditions: a condition where the gender of the musician was known (before the introduction of blind auditions) and a condition where the gender of the musician was unknown (after the introduction of blind auditions). This natural experiment allowed researchers to assess whether women are more likely to be hired when their gender is unknown.*

When dealing with sensitive topics like discrimination or prejudice, asking individuals about their attitudes is less reliable than covertly measuring their behaviors because of which concept?

Social desirability bias *Attitudinal measures are susceptible to social desirability bias, where participants answer questions in socially desirable ways rather than giving truthful responses. For example, a person might mind having a black neighbor more than she would admit on a survey. An employer might respond more positively to an attitudinal measure about his willingness to hire a person with a criminal record compared to what he would do if the actual situation presented itself, as found by Pager and Quillian.*

If a relationship between an independent variable and a dependent variable is ____, it cannot possibly be a causal relationship.

Spurious *Spurious relationships are fully explained by a third variable. Thus, there is no actual association between the two concepts, meaning it cannot possibly be a causal relationship. Experiments uniquely solve the problem of spuriousness. By experimentally holding other factors constant and varying only the independent variable, we are assured that the relationship between the independent variable and the dependent variable is not spurious.*

Which of the following statements about survey experiments is true?

Survey experiments have a higher degree of external validity than laboratory or field experiments *Because survey experiments are conducted on a random sample of the population, they have a higher degree of external validity than laboratory or field experiments, but their internal validity tends to be lower due to less control over other features of the experimental setting. Because participants are drawn from a large, diverse population, they are often spread out geographically. Survey experiments also rely on self-reports of how people believe they would behave or feel, which does not always correspond to actual behavior or feelings.*

What is the third stage of designing a laboratory experiment?

The researcher devises a valid and reliable measure of the dependent variable *In general, a laboratory experiment progresses in four stages: First, the researcher creates a setting that is engaging and makes sense to the participant. Second, the researcher manipulates the independent variable while holding other factors constant. Third, the researcher devises a valid and reliable measure of the dependent variable. Finally, the researcher assesses the quality of the experiment and works to ensure the participant's well-being.*