Research Methods Chapter 10: Experimentation and Validity

Addressing Experimenter Expectancy Effects

(Page 311)

Statistical Conclusion Validity

*Concerns the prior statistical treatment of data and the soundness of the researchers statistical conclusions*. The key question boils down to: When the researchers concluded that there was or was not a statically significant (i.e., non-chance) relation between the independent and dependent variables, was this conclusion based on appropriate statistical analyses? Scientists rely on their experience and knowledge of accepted practices in their field to guide their approach to statistical analysis, and they may consult with expert statisticians. When scientists submit research reports to peer-reviewed journals, the reviewers judgements about statistical conclusion validity will factor heavily into the decision whether the report should be accepted for publication. If reviewers believe the statistical analyses were inadequate, but that in other respects the study had scientific merit, they may recommend that the researchers revise or perform additional statistical analyses and resubmit the report for publication.

Other Approaches to Addressing Demand Characteristics:

- Increasing the psychological realism of the experiment, so that participants will be more involved in the situation and thus more likely to behave spontaneously. - Pilot testing the experiment to help identify potential demand characteristics ahead of time. - Using dependent measures that are unobtrusive or more difficult for participants to distort. - Avoiding within-subject designs when participants exposure to all conditions will likely increase their awareness of the hypothesis. - Identifying participants who do versus don't claim to have been aware of the hypothesis and analyzing their results separately to gain insight into whether this knowledge affected their responses. - Manipulating participants knowledge of the hypothesis to determine whether this affects their responses on the dependant variables.

Debriefing

A conversation with the participant that conveys additional information about the study.

Wait-List Control Group

A group of randomly selected participants who do not receive a treatment, but expect to and do receive it after treatment of the experimental group(s) ends.

Masking (also called Blinding) Approach

A procedure in which the parties involved in an experiment are kept unaware of the hypothesis being tested, and/or the condition to which each participant has been assigned.

Replication and Extension (also called Replication with Extension)

A replication that adds a new design element to the original study.

Quasi-Experiment

A study that has some features of an experiment, but lacks key aspects of experimental control.

Statistical Conclusion Validity: Statistical Issues *Quantitative Psychology*

A subfield of psychology that specializes in issues concerning research design, the measurement of variables, statistical analysis, and mathematical models of behaviour.

Pilot Study

A trial run, usually conducted with a smaller number of participants, prior to initiating the actual experiment.

Randomized Controlled Trial (OR Randomized Clinical Trial)

An experiment in which participants are randomly assigned to different conditions for the purpose of examining the effectiveness of an interaction.

How Experiments Address Threats to Internal Validity: Example 1: Colour and Achievement Performance: *Instrumentation*

As long as random assignment or proper counterbalancing procedures are used, then any instrumentation effects that might occur over the course of an experiment should, overall, affect participants in all conditions to an equivalent degree. Instrumentation is unlikely to be a confounding variable.

How Experiments Address Threats to Internal Validity: Example 1: Colour and Achievement Performance: *Attrition*

Attrition can threaten the internal validity of a well-designed experiment. Although we were able to assume that random assignment created equivalent groups at the beginning of our experiment, differential attrition can result in non-equivalent groups at the beginning of our experiment. Experimenters should determine why participants discontinue and examine any available pretest scores to determine whether continuing versus discontinuing participants differ, overall. Realize that in many laboratory experiments, attrition may be minimal or nonexistent.

Construct Validity: Example 1: Heredity and Learning Ability: Construct Validity of Inference

Because Tryon also used maze performance to infer cognitive ability, we must also consider construct validity of this inference as well. His labels of "bright" and "dull" go to the heart of the matter. You can see that the question of construct validity concerning his independent variables boils down to the following: "*What underlying attribute(s) did maze performance really measure?*". Did heredity influence the rats cognitive ability, or did it instead influence their sensory capabilities, motivational traits, or emotionality?

How Experiments Address Threats to Internal Validity: Example 2: Psychotherapy for Depression: *Selection*

Because participants were randomly assigned to conditions, the therapy and control groups are assumed equivalent, overall, at the start of the experiment.

How Experiments Address Threats to Internal Validity: Example 2: Psychotherapy for Depression: *Instrumentation*

Because the same psychological inventory was used for the pretest and post-test, instrumentation is not a plausible confound regarding that measure. As for the psychologist who observed and rated the participants, pretest and post-test differences in experience, fatigue, and rating criteria could cause instrumentation effects. However, if proper procedures are used (e.g., observations of control group and therapy group participants are randomly alternated; the observer is kept unaware of who is in each group), instrumentation effects should be equivalent in both conditions.

Other Issues Concerning Experimental Control

Beyond designing their research to minimize the aforementioned threats to internal validity, behavioural scientists must also anticipate other potential problems that, if left uncontrolled, could become confounding variables.

Categories of Inference

Concerns about validity apply to different kinds of inferences that scientists make. Strictly speaking, validity applies to inferences about studies and findings; not to the studies or findings themselves. 1. Inferences about constructs 2. Statistical Inferences 3. Causal Inferences 4. Inferences about generalizability

Ecological Validity

Concerns the degree to which responses obtained in a research context generalize to behaviour in natural settings. Some psychologists view it at a subset of external validity. That is, the question of external validity asks whether research conclusions are generalizable to other settings, and this includes both other laboratory and real-life settings. Ecological validity is also often discussed in reference to how well the research setting and methods correspond to what people encounter in daily life.

Internal Validity

Concerns the degree to which we can be confident that a study demonstrated that one variable had a causal effect on another variable. In other words, in an experiment, can we conclude that it truly was exposure to the different conditions of the independent variable, rather than some other factor, that caused the differences in the dependent variable? Inferences about causality have internal validity when the research design and experimental procedures are sounds and thus enable us to rule out plausible alternate explanations for the findings. Poor internal validity results from the presence of confounding variables that provide a reasonable alternative explanation for why participants responses differed, overall, across the various conditions of the experiment.

External Validity

Concerns the generalizability of the findings beyond the present study such as generalization across populations, across settings, and across species (for nonhuman animal research).

Basic Threats to Internal Validity: *Testing*

Concerns whether the act of measuring participants responses affects how they respond to subsequent measures. Thus, we can't rule out the possibility of a testing confound. Changes in motivation or practice habits that occur in response to taking a performance pretest also illustrate testing effects.

Critical Thinking, Inference, and Validity

Conducting research is a grand exercise in critical thinking. Difficult decisions have to be made at many points along the way: what variables to study, how to operationalize them, how to best analyze the data, and so forth. The critical thinking at each juncture often centres on issues of valid inference. Given the feasible options available to us as we plan and conduct our study, which ones will maximize our ability to make valid claims when we report the findings? Concerns about valid inference also go to the heart of evaluating claims encountered in everyday life.

Suspicion Probes

Conversational strategies conducted during debriefing in which experimenters explore participants beliefs about the study and its hypothesis. They are probably *the most common approach to addressing whether demand characteristics influenced participants behaviour*. For them to be effective, the experimenter first needs to establish rapport with participants during debriefing, introduce the probes gradually, begin probing prior to revealing the true hypothesis, and progressively pursue the participants beliefs in greater depth if they initially claim that they weren't aware of the hypothesis. Particularly before rapport is established, asking participants direct questions about whether they knew the hypothesis may elicit untruthful answers from many participants who did, in fact, know or figure out the hypothesis.

Yoked Control Group

Each control group member is procedurally linked (i.e., yoked) to a particular experimental group member, whose behaviour will determine how both of them are treated. Typically this is done through random assignment or matching.

Single-Blind Procedure

Either the participants or experimenter, but not both, are masked to the participants condition.

How Experiments Address Threats to Internal Validity: Example 1: Colour and Achievement Performance

Elliot and coworkers exposed college students to the colour red, green, or black during one laboratory session per participant. Those exposed to red subsequently performed most poorly on an anagram task.

How Experiments Address Threats to Internal Validity: Example 1: Colour and Achievement Performance: *Regression to the Mean*

Elliot et al. did not select participants based on extreme scores. Even if they had, the degree of regression to the mean should be equivalent across all conditions as long as participants are randomly assigned. This eliminated regression to the mean as a plausible confounding variable.

Establishing Generalizability

Evidence for or against external validity accrues over time as scientists replicate and build on the original research. No study will be able to satisfy all questions about external validity, but even in their initial research on the topic experimenters can take some steps to increase confidence in the external validity of their findings. For example, scientists can replicate their own research within their initial research project.

Conceptual Replication

Examines the same question investigated in the original study, but operationalizes the constructs differently.

How Experiments Address Threats to Internal Validity: Example 1: Colour and Achievement Performance: *Maturation*

Experiments don't prevent maturation, but by randomly assigning participants to conditions, Elliot et al. could assume that any maturation effects would be equivalent across the various conditions. Therefore, maturation is not a plausible confounding variable.

How Experiments Address Threats to Internal Validity: Example 1: Colour and Achievement Performance: *Selection*

Experiments involve multiple conditions, and when between-subjects deigns are used, the key to preventing a selection confound is to create equivalent groups at the start. Experimenters achieve this by randomly assigning participants to conditions, as Elliot et al. did.

Types of Validity

Four types of validity that are of central concern when conducting experiments: 1. Construct Validity 2. Statistical Conclusion Validity 3. Internal Validity 4. External Validity

How Experiments Address Threats to Internal Validity: Example 2: Psychotherapy for Depression: *History*

General history effects cannot explain the findings. If the economy improved, if the seasons changed, if world peace broke out during the study, all participants would be exposed to those events. Moreover, because participants were randomly assigned, in principle there is no reason to assume that significant overall history differences will exist regarding personal events ( e.g., vacations, job changes, family births).

Construct Validity

In the contract of experimentation, construct validity applies to both measuring and manipulating variables. Specifically, it concerns the issue of whether the constructs (the conceptual variables) that researchers claim to be studying are, in fact, the contracts that they truly are manipulating and measuring. It is affected by how faithfully the operational definitions of the independent and dependant variables represent the contracts that the researchers intend to study. The operations in one experiment are rarely pure representations of contracts.

Complete Replication (also called Full Replication)

Includes all the conditions of the original study.

Partial Replication

Includes only some of the original conditions of the original study.

Good Subject Role

Involves providing responses that help to support the perceived hypothesis of the study. This norm partly arises from peoples hope that their response will contribute to science and the study's success and usually occurs unconsciously. Although, some participants may assume a defiant-subject role and act in ways that will disconfirm the perceived hypothesis, in general the good-subject role seems to prevail.

Basic Threats to Internal Validity: *Regression to the Mean*

Is the statistical concept that when two variables are not perfectly correlated, more extreme scores on one variable will be associated overall with less extreme scores on the other variable.

Construct Validity: Example 1: Heredity and Learning Ability

Its one thing to claim that, "Selective breeding produced differences in maze performance errors between two strains of rats". Its quite another to claim that hereditary (genetics) produced differences in the rats cognitive ability. The first statement is a *causal inference* at an operational level; the second, a causal inference at a conceptual level. *Construct validity concerns the validity of the leap from the operational to the conceptual level*. Tryon considered several operational definitions of "maze learning ability": number of errors, amount of time to reach the end of the maze, number of trials needed to run the maze without making any errors, and an overall score of combining all factors. He examined the properties of each measure and decided that, "*the number of errors*" was the most valid one.

How Experiments Address Threats to Internal Validity: Example 1: Colour and Achievement Performance: *Testing*

Many experiments do not include a pretest because due to random assignment, the participants in various conditions are, overall, assumed to be equivalent at the start of the experiment. This eliminates the possibility of a testing confound. Elliot et al.'s experiment did involve a pretest, but because all participants took it, testing effects should be equivalent in all the conditions and therefore won't be a confounding variable.

Manipulation Checks

Measures to assess whether the procedures are used to manipulate and independent variable successfully captured the construct that was intended.

Double-Blind Procedure

Neither the participants nor the experimenters are aware of who is receiving the actual treatment and who is receiving the placebo.

How Experiments Address Threats to Internal Validity: Example 2: Psychotherapy for Depression: *Attrition*

No matter how well the study is designed, attrition remains a concern. If attrition is high or if differential attrition occurs, Dr. Rodriguez should try to asses whether participants in the two conditions discontinued for different reasons, and decide whether attrition is a plausible alternate explanation for the results.

Basic Threats to Internal Validity: *Attrition* (OR *Subject Loss*)

Occurs when participants fail to complete a study. It can occur for many reasons and is always undesirable. *It poses the greatest threat to internal validity when participants who discontinue differ from those who complete the study in some attribute that could account for the changes obtained on the dependent variable*.

Ceiling Effect

Occurs when scores on a dependant variable bunch up at the maximum score level.

Floor Effect

Occurs when scores on a dependant variable bunch up at the minimum score level.

Differential Attrition

Occurs when significantly different attrition rates for discontinuing exist, overall, across the various conditions. It suggests that something intrinsic to certain conditions caused attrition and may bias the results.

Placebo Control Group

Participants do not receive the core treatment, but are led to believe that they are (or may be) receiving it.

Placebo Effect

Peoples expectations about how a treatment will affect them influence their responses (on the dependent variable) to that treatment.

How Experiments Address Threats to Internal Validity: Example 2: Psychotherapy for Depression

Pretest depression scores are used to identify severely depressed adults, who are then randomly assigned to therapy or control conditions. Different types of control conditions could be used of which several of them may be used. The findings indicate that, overall, participants in the two conditions do not differ significantly in level of depression on the pretest, but on the post-test those who received therapy were significantly less depressed than the control group participants.

Experimentation and Validity: Claims About Absolute Proof

Recall that scientists typically avoid claims about absolute proof when discussing research results.

Range Restriction

Reduced variability in the scores on a dependant measure. Examples include the ceiling and flow effect of which can affect both correlational studies as well as experiments.

Basic Threats to Internal Validity: *Instrumentation*

Refers to changes that occur in measuring instrument during the course of data collection. When observers measure behaviour, systematic changes in their performance during a study represent instrumental effects. Such effects could result from factors such as observer illness or fatigue, gaining experience with a rating or coding system, and non-consciously adopting different criteria when deciding how to rate or categorize a response.

Demand Characteristics

Refers to cues that influence participants beliefs about the hypothesis being tested and the behaviours expected of them. If demand characteristics lead participants to guess the hypothesis accurately, this may create a plausible alternative explanation if the hypothesis is supported: Were participants responses influenced by the intended manipulation of the independent variable, by demand characteristics, or by both?

Red Herring Technique

Refers to diverting peoples attention from a real issue by raising an irrelevant issue. In this approach of addressing demand characteristics, researchers tell participants a fictitious story about the experiments purpose and also create misleading demand characteristics to divert the attention of any participants who may still be trying to figure out what the experiment is really about. This procedure can be effective, but has the *ethical drawback of adding another layer of deception*.

Basic Threats to Internal Validity: *History*

Refers to events that occur while a study is being conducted, and that are not part of the experiment manipulation or treatment. Whether history rises to the status of a plausible confounding variable will depend on the events that took place during the period of the study.

Basic Threats to Internal Validity: *Selection*

Refers to situations in which at the start of the study, participants in the various conditions already differ on a characteristic that can partly or fully account for the eventful results.

Sensitivity

Refers to the ability to detect an effect that actually is present.

Replication

Refers to the process of repeating a study in order to determine whether the original findings will be upheld.

Basic Threats to Internal Validity: *Maturation*

Refers to the way the people naturally change over time, independent of their participation in a study. This includes changes in cognitive and physical capabilities that occur with aging, fluctuations in alertness and fatigue that accompany biological rhythms, and normal recovery from physical illness or psychological disorders. Maturation also includes the general accrual of knowledge and skills as we gain more experience over time.

Psychological Realism

Represent the degree to which the experimental setting is made psychologically involving for participants, thereby increasing the likelihood that they will behave naturally rather than self-monitor and possibly distort their responses.

Basic Threats to Internal Validity

Seven Sources of Threat: 1. History 2. Maturation 3. Testing 4. Instrumentation 5. Regression to the Mean 6. Attrition 7. Selection

Statistical Conclusion Validity: Claim Issues

Statistical Conclusion Validity allows us to claim that there is an association between the independent and dependent variables, and that its unlikely this relation is due merely to chance (i.e., to random variations in behaviour). But, statistical conclusion validity alone does not enable us to conclude that the relation is causal. To conclude that the independent variable was responsible for producing the changes in the dependent variable, we need to be confident that other factors were not responsible.

Inference in Tryon's Selective Breeding Project

The goal of the experiment was to demonstrate experimentally that individual differences in psychological qualities, such as learning ability, had a hereditary basis. This research helped to establish a new field of study called *behaviour genetics*. Tryon did not claim in his research reports that he was examining the genetic basis of intelligence. He believed that there was an "extensive universe" of more specific behavioural competencies and believed that maze running involved mental skill. He concluded that his bright and dull rats differed in cognitive ability as a function of their experimentally controlled hereditary.

How Experiments Address Threats to Internal Validity: Example 1: Colour and Achievement Performance: *History*

The highly controlled laboratory environment and short duration of each experimental session minimized each students exposure to extraneous external events while being studied. By using a procedure such as *block randomization* to assign participants to the various conditions, the potential influence of such history effects should be distributed equivalently across those conditions. History can be a problem if an experiment is poorly executed.

Mundane Realism

The surface similarity between the experimental environment and real-world settings

Statistical Conclusion Validity: Statistical Issues: *Violations of Statistical Tests*

There is a longstanding debate among statistical experts about whether some frequently used statistical tests are "robust" to certain types of violations. *Robust* means that a statistical test can yield accurate results (or that the amount of error will only be slight) even if a data set violates the tests statistical assumptions. However, even if one sides with experts who argue that these tests are robust about certain violations, violating other assumptions of these tests can severely compromise statistical conclusion validity.

How Experiments Address Threats to Internal Validity: Example 2: Psychotherapy for Depression: *Maturation, Regression to the Mean, and Testing*

These are not plausible confounding variables. Participants came from the same population (severely depressed adults) and were randomly assigned. Overall, the participants in therapy and controlled conditions should experience equivalent maturation (e.g., spontaneous remission). On average, changes in pretest and post-test depression scores in the two conditions should reflect and equivalent degree of statistical regression. As for testing, all participants took the psychological inventory twice and overall should experience equivalent testing effects.

Construct Validity: Example 2: Colour and Achievement Performance

They measured the construct of avoidance motivation by recording students psychological responses and giving them a choice to work on an easy or moderately difficult task. Selection of the easy task was taken as evidence for avoidance motivation. Are these valid measures? Fortunately, prior research had already established the construct validity for these measures. This is one example of how researchers build on the with of scientists who came before them.

Minimizing Response Biases

To minimize this response bias, experimenters typically conceal the hypothesis and study's specific purpose from participants until the debriefing session. Participants will assume that the experimenter is "looking for something" and form beliefs about the hypothesis. Research settings provide cues the may shape participants guesses about the hypothesis. These cues can include the experimenters behaviour, a laboratory's layout, and the nature of the experimental tasks.

Construct Validity: Example 1: Heredity and Learning Ability: Tryon's Findings

Tryon concluded that the main difference between the bright and dull rats lay in a specific cognitive ability to form spatial orientations. Controversy continued, however, over whether Tryon's rat strains primarily differed in cognitive or non-cognitive attributes, and whether he had assessed a highly specific or more general maze-learning ability. In any case, his rigorous research program provided some of the first sound scientific evidence that heredity influenced a species psychological characteristics. Among the findings: it appears that maze-bright rats have a stronger liking for alcohol than do maze-dull rats. Do not, however take this as evidence for supporting the sorely mistaken belief that drinking makes you smarter!

Experimenter Expectancy Effects

Unintentional ways in which researchers influence their participants to respond in a manner consistent with the researchers hypothesis.

Statistical Conclusion Validity: Statistical Issues: *Inferential Statistical Tests*

Used for determining statistical significance typically require that certain assumptions be met in order for a particular test to be used in a valid manner. For example, the proper use for a statistical test assumes that there is a certain minimum number of observations in each cell of a research design. Another assumption involves the scale of measurement. Some statistical tests are designed for use with variables on an interval or ratio scale, other tests can be used for variables measured on an ordinal scale, and others for variables measured on a nominal scale. If a researcher uses a statistical test when the requisite statistical assumptions are violated, statistical conclusion validity will be compromised.