Experimental Week 9: Experiments and Internal Validity
Single-blind Study
A study in which the participants are unaware of whether they are in the control group or the experimental group or a study in which the subjects do not know if they are in the experimental or the control group. - When participants receive a placebo, if only the participants are unaware of their condition, this is known as a single-blind study.
Instrumentation (threat to internal validity)
A threat to internal validity only because repeated measurements have changed the quality of the measurement instrument. A threat to internal validity that occurs when a measuring instrument changes over time from having been used before. Changes occur in a maturation (change going on in participants behavior over time) but change is caused by the testing procedure itself as maturation is due to a natural change regardless. - A change in the measuring device during the course of a study can be a confound if the instrument is poorly calibrated or malfunctions, especially if the unreliable device is used more often in one condition than in another. - Ex: Two research assistants were trained with the same protocol. One assistant measured the behaviors of the experimental group, and the other assistant measured the behaviors of the control group. Reaction time was measured for the control group and the experimental groups. Time to the millisecond was measured for both groups, but a stopwatch was used for one group and a computerized application was used for the other. Both choices that involve using different forms of instrumentation for the experimental and control groups, in this case different observers and different devices, pose an instrumentation threat to internal validity. It is possible that the observer for one group is measuring things differently or the device used for one group is not working properly. The same devices and observers should be used for both the experimental and control groups.
Testing (threat to internal validity) (The pre-test influences the outcomes of the post-test.)
A threat to internal validity that occurs when taking the pretest affects how participants do on the posttest. Measuring participants response more than once, they tend to respond differently on subsequent measures, and it leads to Practice or Fatigue Effect. The tendency of participants who are tested multiple times to be influenced by this "practice" in a way that changes their performance in subsequent tests. If participants are tested multiple times, then practice or repetition of test items, tasks, or activities can influence participants' performance in subsequent tests. - Practice Effect: Getting better over time, it will lead to an increase of scores due to your intervention or practice you are getting. - Fatigue Effect: Getting fatigue and getting worse over time and being more aware. - Ex: A researcher is measuring the ease of using a new university app to gain information about course schedules and campus information. The researcher has participants complete a pretest when they are first exposed to the app. Then she has the participants attend a seminar on how to use the app efficiently and complete the original measure again as a posttest. She finds that participants showed more improvement on the posttest, that is they found more information and completed the tasks faster; she concludes that the seminar must be responsible for the change. (Answer = Testing Effect. Participants may perform better on the posttest because of their prior exposure through repeated testing and not because of the seminar, which is an intervention.) - Ex: Participants showed higher productivity at the end of the study because the same test was administered. Due to familiarity, or awareness of the study's purpose, many participants achieved high results.
Maturation (threat to internal validity) (The outcomes of the study vary as a natural result of time.)
An experimental gap improves over time only b/c of natural development spontaneous improvement or effects related to the passage of time, such as aging. People change over time in terms of their behavior or performance regardless of what happens to them during a study, don't be confused that change is due to manipulation of IV. Naturally occurring time-related changes in participants. - Sometimes, naturally occurring time-related changes in participants—called maturation—can influence the outcome of a study, thereby confounding it. This is more likely to occur in longitudinal studies (i.e., studies that take place over a period of time, sometimes several years). - Ex: A researcher measured depression symptoms in a group of college freshmen and conducted a follow-up study two years later when the students were juniors. She found that depression symptoms decreased for nearly all students and attributed the change in symptoms to the counseling interventions that were provided. (Answer = Maturation. The participants could naturally have fewer symptoms over time, without the intervention. Typically, anxiety and depression would be higher during the first year of college than the third year. - Ex: Most participants are new to the job at the time of the pre-test. A month later, their productivity has improved as a result of time spent working in the position.
History (threat to internal validity) (An unrelated even influences the outcomes)
An experimental group changes over time because of an external factor or event that affects all or most members of the group. A threat to internal validity that an experimental group changes over time because of an external factor or event that affects all or most members of the group. Events occur during a study that are unrelated to the experimental manipulation. A threat to internal validity that an experimental group changes over time because of an external factor or event that affects all or most members of the group. The tendency of events or circumstances outside an experiment to influence the outcome, particularly in pretest-posttest studies. - Ex: A week before the end of the study, all employees are told that there will be layoffs. The participants are stressed on the date of the post-test, and performance may suffer. - Ex: A researcher designs a pretest-posttest study to measure test-taking time before and after participants learn a technique designed to promote more efficient, which means faster, test taking skills in a data analytics course. The control group, in a different section of the data analysis course but with the same instructor, completes the pretest and posttest, but does not learn the new technique. (Answer = Because the groups are taking the same course, it is possible they have other classes together as well. If so, imagine that they are all exposed to a test-taking workshop in a different class. The techniques learned are different from those in the study, but actually more effective.)
Regression to the mean (threat to internal validity)
If the first measurement is extreme, second measurement will be closer to the mean or the tendency of extreme scores on a variable to be followed by, or associated with, less extreme scores. The tendency of extremely high or extremely low scores to become more moderate (i.e., closer to the mean) with repeated measurement of the dependent variable. - Statical phenomena where participants who score extremely high or poorly being tested again, and they then regress towards the average. Those who receive scores tends to have fewer extreme scores when retested even in the absence of any treatment effects. It is attributable to measurement error. - Extremely high or extremely low scores naturally tend to become more moderate (i.e., closer to the mean) with repeated measurement of the dependent variable. That is, if a participant scores very high on a measure at Time 1, then at Time 2, he is likely to score lower on the same measure. Thus, if a researcher selects participants on the basis of their extreme pretest scores, then shifts in these participants' scores during a study may occur because of regression to the mean rather than any treatment or experimental manipulation. - Ex: An instructor provides a three-point true/false in-class quiz during each class session. She begins to think the quizzes are too easy because a large number of students have a grade of 100% after the first four quizzes. She decides to add more questions to increase the difficulty. At the next progress check, comprised of nine quiz grades, she is pleased to see that the average has dropped to 85% and is convinced she has arrived at the appropriate level of difficulty by adding more questions. (Answer = Regression to the Mean. The initially high scores may have been due to chance. It is common for high scores to drift back toward the mean over time. So, the change in the quiz scores, with a 50% chance of selecting the correct answer, may not be due to the added questions.) - Ex: Sport illustration cover jinx where performance goes done after the cover, but this is due to a natural regression to the mean as it is difficult to maintain high performance.
What is a confounding variable?
In an experiment, a factor other than the independent variable that might produce an effect. A factor other than the independent variable that might produce an effect in an experiment. - Extraneous variable: Factors, other than the IV that are controlled by the experimenter because they have to potential to influence the DV and change the results of the experiment. (Ex: People pre-existing problem of whether they can solve a sudoku problem could impact my study.) - In experimental research, a confounding variable is a variable other than the independent variable that causes a change in the dependent variable. Confounding variables (sometimes called confounds) are particularly troublesome because they act directly on the dependent variable, making it hard to conclude definitively that manipulation of the independent variable caused changes in the dependent variable. - What do confounding variables have to do with internal validity? An experiment is high in internal validity if it provides compelling evidence that only manipulation of the independent variable—and not the action of other confounding variables—caused observed changes in the dependent variable. When we evaluate the internal validity of a causal claim - Uncontrolled extraneous variables that it does impact our study, when we fail to control, and we have no way of knowing whether the confound variable or IV caused the change in the DV.
Selection bias (threat to internal validity)
In an experiment, unintended differences between the participants in different groups. Comparison groups are not equal at the beginning of the study or a partiality in choosing the participants in a study. - At the start of the experiment, groups are non-equivalent, and this difference affects the results. Happens when you have small groups, not using random assignment or using a pre-formed group. - Ex:
Define the common threats to internal validity and be able to identify them when you are given a description of a study (history, maturation, instrumentation, testing/order, regression to the mean, selection bias).
Internal Validity is the extent to which we can draw cause-and-effect inferences from a study or the degree to which changes in the dependent variable are due to the manipulation of the independent variable. An experiment is high in internal validity if it provides compelling evidence that only manipulation of the independent variable—and not the action of other confounding variables—caused observed changes in the dependent variable. When we evaluate the internal validity of a causal claim, we are ensuring that the study meets Mill's criteria for establishing causality. - History Threat: An experimental group changes over time because of an external factor or event that affects all or most members of the group. A threat to internal validity that an experimental group changes over time because of an external factor or event that affects all or most members of the group. Events occur during a study that are unrelated to the experimental manipulation.
Double-blind Study
Study in which neither the experimenter nor the subjects know if the subjects are in the experimental or control group or an experiment in which neither the participant nor the researcher knows whether the participant has received the treatment or the placebo. - If both the participants and the researchers are unaware of the participants conditions, it is referred to as a double-blind study.
What is the Placebo Effect?
The phenomenon in which the expectations of the participants in a study can influence their behavior. - If participants have expectations of the effects of an experimental condition (including the effects of a medication), their expectations will confound the study. For example, sick people who take a sugar pill (i.e., a pill with no active ingredient to treat the sickness) may report feeling better. This effect is known as the placebo effect, and it can threaten internal validity if only the experimental group (or treatment group) expects to see improvement. - To ensure that both the experimental group and control group have the same expectations, the control group receives a placebo—a therapeutically inert substance or nonspecific treatment—without knowledge of whether they are in the treatment or control condition.
Why are participant characteristics not considered true independent variables?
These are the characteristics that make people unique: age, sex, ethnicity, education, socioeconomic status, marital status, living arrangements, employment status, personal habits, personality traits, and so on. Participant variables cannot be manipulated in experiments. A researcher cannot, for example, manipulate a participant's age; she cannot make a participant older or younger. Therefore, participant variables are never independent variables. However, they can still affect the dependent variable, and so they are a potential source of confounding.
What's the point of controlling an extraneous variable?
To prevent them from becoming a confounding variable. - In order to infer a cause-and-effect we only want to manipulate the IV and control extraneous variables are we able to infer a cause-and-effect relationship in experimental research only. - Manipulation demonstrates the direction of all extraneous variables ensure that only the IV (no other factors) is affecting the DV.
Mortality (threat to internal validity)
the loss of study subjects
