Research Methods 2.0
independent-groups design
An experimental design in which different groups of participants are exposed to different levels of the independent variable, such that each participant experiences only one level of the independent variable. Also called between-subjects design, between-groups design EX: two different groups each use different notetaking method
matched groups
An experimental design technique in which participants who are similar on some measured variable are grouped into sets; the members of each matched set are then randomly assigned to different experimental conditions. Also called matching
Interpreting output
You should be able to interpret the following types of output from SPSS: correlation matrix, reliability analysis (Cronbach's Alpha), you have examples of some of these in your output for lab 1
observer bias,
researchers influence results Solved by: double blind study
statistical validity
statistical significance, effect size, replications Are statistical conclusions accurate Do numbers support claim? Central to evaluating all 3 types of claims
nderstand how to interpret t-test results
statistical significance: the degree to which a research outcome cannot reasonably be attributed to the operation of chance or random factors Indicated by a critical probability (p) value- = the chances that there is actually no difference between groups In psych, the critical p usually .05
interrater reliability
the degree to which different observers agree on their observations Correlations or % agreement if 2 raters Correlation should be .7 or higher, % agreement close to 100%
criterion validity
the extent to which a measure is related to an outcome associated with specific related behavior or outcome objective - correlation analyses e.g., SAT math score predict math GPA in College
content validity
the extent to which a test samples the behavior that is of interest based on comprehensive description of content math test has math problems representing various areas of mathematics subjective, systematic: requires thorough knowledge of construct
correlation coefficient
the measures of the relationship between two variables
theory-data cycle
theory-> research questions->research design-> preregistered hypothesis->data
test-retest reliability
using the same test on two occasions to measure consistency Only good for stable variables . Assessed with correlation coefficient At least .5, with stronger correlations for shorter time periods between tests
response set
A shortcut respondents may use to answer items in a long survey, rather than responding to the content of each item. Also called nondif erentiation.
pilot study
A study completed before (or sometimes after) the study of primary interest, usually to test the effectiveness or characteristics of the manipulation
One-group pretest/posttest design
An experiment in which a researcher recruits one group of participants; measures them on a pretest; exposes them to a treatment, intervention, or change; and then measures them on a posttest.
demand characteristics
participants figure out the hypothesis Solved by: double blind study
observational measures
pro: Avoids memory con: Tough to observe some B & observing may alter B
9 threats to internal validity Understand the definitions of each threat and be able to recognize examples
1. History 2. Maturation 3. Testing 4. Instrumentation 5. Statistical regression 6. observer bias, 7. demand characteristics 8. placebo effects 9. attrition threats
confound
A general term for a potential alternative explanation for a research finding; a threat to internal validity.
forced choice questions
A survey question format in which respondents give their opinion by picking the best of two or more options
open ended questions
A survey question format that allows respondents to answer any way they like
likert scale
A survey question format using a rating scale containing multiple response options anchored by the specific terms strongly agree, agree, neither agree nor disagree, disagree, and strongly disagree
design confound
A threat to internal validity in an experiment in which a second variable happens to vary systematically along with the independent variable and therefore is an alternative explanation for the results.
regression threats
A threat to internal validity related to regression to the mean, a phenomenon in which any extreme finding is likely to be closer to its own typical, or mean, level the next time it is measured (with or without the experimental treatment or intervention). EX: There was an unusually high level of drinking when the first measurement took place Solved by: Comparison group - no RDC
selection effect
A threat to internal validity that occurs in an independent-groups design when the kinds of participants at one level of the independent variable are systematically different from those at the other level.
instrumentation threats
A threat to internal validity that occurs when a measuring instrument changes over time EX During the first measurement students were asked "What is the largest number of drinks you had during a single day last week" , but during the second measure they were asked "how many drinks did you average per day last week Solved by: Maintain equality in measurement tools
maturation threats
A threat to internal validity that occurs when an observed change in an experimental group could have emerged more or less spontaneously over time. EX: Students were all 8 months older when measured the 2nd time, they had settled down and adjusted to college life Solved by: Comparison group - no RDC
history threats
A threat to internal validity that occurs when it is unclear whether a change in the treatment group is caused by the treatment itself or by an external or historical factor that affects most members of the group. EX: a student alcohol related death may have changed students attitudes toward drinking; Police, bars & liquor stores may have started cracking down on fake IDs; final exams Solved by: Comparison group - no RDC
practice effects
A type of order effect in which participants' performance improves over time because they become practiced at the dependent measure (not because of the manipulation or treatment). Also called fatigue effect
carryover effect
A type of order effect, in which some form of contamination carries over from one condition to the next.
double-barreled questions
A type of question in a survey or poll that is problematic because it asks two questions in one, thereby weakening its construct validity.
leading questions
A type of question in a survey or poll that is problematic because its wording encourages one response more than others, thereby weakening its construct validity.
external validity
Do results generalize to or represent populations and/or settings outside of the study Relevant to all, most important in frequency claims
covariance
Do the results show that the variables are correlated? The degree to which two variables go together. Also one of three criteria for establishing a causal claim, which states that, in a study's results, the proposed causal variable must vary systematically with changes in the proposed outcome variable.
temporal precedence
Does the method establish which variable came first in time? One of three criteria for establishing a causal claim, stating that the proposed causal variable comes first in time, before the proposed outcome variable
physiological measures
pro: Tough to fake, assess things outside awareness con: expensive and time consuming
socially desirable responding
Giving answers on a survey (or other self-report measure) that make one look better than one really is. Also called faking good
Really bad experiment example
Imagine that you are a counselor at a college and you're trying to decrease student drinking. You've read that a program called Responsible Drinking on Campus (RDC) might be effective. You Decide to implement the program to reduce drinking on your campus but you want to collect data to show that it works. So you measure drinking during the first week of classes in the fall (before the program starts), with an anonymous self report item asking,"how many drinks have you had during the last week". You then run the program (required information sessions for 5 weeks in the middle of both terms). Then you measure drinking behavior again in the spring a week before classes end. After analyzing the data, you find that campus drinking decreased by 33% after implementation of the program.
attrition threats
In a pretest/posttest, repeated-measures, or quasi-experimental study, a threat to internal validity that occurs when a systematic type of participant drops out of the study before it ends. EX: The really heavy drinkers flunked out or were expelled Solved by: Remove dropout scores from pretest average
testing threats,
In a repeated-measures experiment or quasi-experiment, a kind of order effect in which scores change over time just because participants have taken the test more than once; includes practice effects. EX: Reporting how much they were drinking the first time made the heavy drinkers realize they were drinking to much so they cut back Solved by: Comparison group
order effects (practice effects, carryover effects)
In a within-groups design, a threat to internal validity in which exposure to one condition changes participant responses to a later condition
control variable
In an experiment, a variable that a researcher holds constant on purpose.
independent variable
In an experiment, a variable that is manipulated. In a multiple regression analysis, a predictor variable used to explain variance in the criterion variable
manipulation check
In an experiment, an extra dependent variable researchers can include to determine how well a manipulation worked
dependent variable
In an experiment, the variable that is measured. In a multiple regression analysis, the single outcome, or criterion variable the researchers are most interested in understanding or predicting.
placebo effects
Participants influenced because they believe the IV does something (even if it actually doesn't) Solved by: double-blind placebo control study
self-report measures
pro: cheap & easy con: Relies on memory, honesty & awarenes
frequency claims
Predicts/describes a particular level or degree of a single variable frequency claims involve only one measured variable EX: 41% Percent of Children Worldwide Experience Moderate Food Insecurity The majority of Mount students support the new smoking ban
reverse-worded items
Survey questions that cause respondents to slow down and think about their answers; intended to weed out yea-sayers and nay-sayers
replication
The process of conducting a study again to test whether the result is consistent
random assignment
The use of a random method (e.g., flipping a coin) to assign participants into different experimental groups.
experiment
a set of controlled observations that test the hypothesis
internal reliability
a study participant gives a consistent pattern of answers, no matter how the researcher has phrased the question Cronbach's alpha Should be .8 or higher
internal validity
a study's ability to rule out alternative explanations for a causal relationship between two variables Central to causal claims only confidence that manipulated variable ( IV) is the actual cause of any changes in measured variable (DV)
within-groups design Understand the advantages & disadvantages of within-groups design
pros: Equivalent groups, Fewer participants con: Order effects Might not be possible Demand characteristics EX: One group uses both note taking methods
associational claims
argues that one level of a variable is associated with a particular level of another variable. supported by studies that have at least two measured variables. Variables that are associated are said to correlate EX: Study Links Coffee Consumption to Lower Levels of Depression in Women Women will be more likely than men to support abortion rights
causal claims
argues that one variable causes changes in the level of another variable
conceptual definition
careful theoretical definition of the construct E.g., a level of proficiency in scholastic work in general or in a specific skill, such as arithmetic or reading.
t-test
differences between means
face validity
extent to which respondents can tell what the items are measuring math test has math problems subjective, superficial
operational definition
how construct is measured or manipulated in study
construct validity
how well constructs were measured and or manipulated. Central to evaluating all 3 types of claims
