Research Methods Ch. 3

External Validity of Frequency Claim Headline

"72% of the world smiled yesterday"

Scatterplot

-A graphical representation of an association -A graph in which one variable is plotted on the y axis and the other variable is plotted on the x axis. -Each dot represents one participant in the study, measured on the two variable

Zero Association (Zero Correlation)

-A lack of systematic association between two variables -EX: The headline "A late dinner is not linked to childhood obesity, study shows" is an example of a zero association, or no association between the variables

Measured Variable: Observed & Recorded

-A variable in a study whose levels (values) are observed and recorded. -EX: height, IQ, gender and hair color, using familiar tools -Psychologists measure more abstract variables such as depression and stress, researchers devise a special set of questions to represent the various levels. -In each case, measuring a variable is a matter of recording an observation, a statement, or a value as it occurs naturally.

Manipulated Variable: Controlled

-A variable in an experiment that a researcher controls, by assigning participants to different levels of that variable (values). -EX: A researcher assigns some people to take a test in a room with other people and assign others to take the test alone. -In both examples, the participants could end up at any of the levels because the researchers do the manipulating, assigning participants to be at one level of the variable or another.

Conceptual Variable (construct)

-A variable of interest, stated at an abstract level, usually defined as part of a formal statement of a psychological theory -Theoretical concepts such as "infant temperament" and "anxiety."; Think of them as being "up in the clouds," but we need to bring them down to the ground from the lofty theoretical level so that we may measure them. -In journal articles, referred to at the conceptual level.

Association Claim

-About two variables, in which the value (level) of one variable is said to vary systematically with the value of another variable -States a relationship between at least two measured variables -Variables that are associated are to said to correlate (relate) -Use verbs such as link, associate, correlate, predict, tie to, and be at risk for. -You will find a correlational study supporting it. -EX: "Study Links Coffee Consumption to Lower Depression in Women". The variables are amount of coffee consumption and level of depression: Higher coffee consumption is associated with lower levels of depression (and therefore lower coffee consumption goes with higher levels of depression).

Variables that can ONLY BE MEASURED—not manipulated

-Age can't be manipulated, it is a naturally occurring variable, researchers can't assign people to be older or younger; they can only measure what age people already are. -IQ is a variable that can't be manipulated, researchers cannot assign some people to have a high IQ and others to have a low IQ; they can only measure each person's IQ. -Sometimes it is unethical to manipulate variables. -EX: In a study on the long-term effects of elementary education, you could not ethically assign children to "high-quality school" and "low-quality school" conditions.

Constant

-An attribute that could potentially vary but that has only one level in the study in question. -EX: A study concluded "15% of Americans smoke," nationality is not a variable because everyone in the study is American. -Nationality would be a constant -Smoking would be a variable, and its levels would be smoker and nonsmoker.

Statistical Validity of Frequency Claims

-An estimate of that value in some population. -EX: The report claiming "39% of teenagers text while driving," researchers interviewed a sample of about 9,000 teen drivers to estimate the behavior of the population of all U.S. teenage drivers. -To evaluate, we start with the point estimate; in a frequency claim, the point estimate is a % -Then we ask about the precision of that estimate; for a frequency claim, precision is captured by the confidence interval (CI), or margin of error of the estimate

Construct Validity

-An indication of how well a variable was measured or manipulated in a study -The extent to which the operational variables in a study are a good approximation of the conceptual variables -How well a conceptual variable is operationalized.

Casual Claim

-Arguing that a specific change in one variable is responsible for influencing the value of another variable -Has two variables, like association claims -Whereas an association claim merely notes a relationship between two variables, a causal claim goes even further, arguing that one of the variables is responsible for changing the other. -EX 1: Pretending to Be Batman Helps Kids Stay on Task; Pretending to be Batman and staying on task -EX 2: Family Meals Curb Eating Disorders; Family meals and eating disorders -The two variables in question covary -EX 3: Eating family meals goes with lower rates of eating disorders. -You might also see a causal claim based on a zero association that reports a lack of cause. -EX 4: You might read that vaccines do not cause autism or that being in daycare does not cause behavior problems. -They use verbs such as cause, enhance, affect, decrease, and change

Examples of Casual Claim Verbs

-Causes -Promotes -Affects -Reduces -May curb -Prevents -Exacerbates -Distracts -Changes -Fights -May lead to -Worsens -Makes -Increases -Sometimes makes -Trims -Hurts -Adds

Variable

-Concepts of interest that vary, form the core of psychological research. -Has at least two levels -Can be measured or manipulated. -In a study can be described in two ways: 1) As Conceptual Variables (elements of a theory) and 2) As Operational Definitions (specific measures or manipulations in order to study them). -EX: The headline: "Most students don't know when news is fake."; "knowing when news is fake" is the variable, and its levels are knowing when news is fake, and not knowing when news is fake.

STATISTICAL VALIDITY OF ASSOCIATION CLAIMS

-Considers how strong the estimated association is and how precise that estimate is -The stronger the association, the more accurate our predictions will be. -Statistical significance: If an association is statistically significant, then the result is probably not due to chance based on that sample -When interrogating the statistical validity of an association claim, there are two kinds of mistakes that can be made. 1) Type I error: false positive; a study might mistakenly conclude that there is an association between two variables in their sample when there actually is no association in the population; you want to increase the chances that you will find an association only when there really is an association. 2) Type II error: a "miss"; a study might mistakenly conclude from a sample that there is no association between two variables when there actually is an association in the population; you want to minimize the chances of missing associations that are really there.

Temporal Precedence

-Criteria for establishing a causal claim -The method was designed so that the causal variable comes first, before the effect variable (outcome variable). -To make the claim "Pretending to be Batman helps kids stay on task," a study must show that "pretending to be Batman" came first and staying on task came later.

Frequency Claim

-Describes a particular rate or degree of a single variable -EX: Most Students Don't Know When News Is Fake; "most" refers to a proportion of students (higher than 50% -Involves only one measured variable, such as level of food insecurity, rate of smiling, or amount of texting. -EX: Measured children's food insecurity using a questionnaire and reported the results. -EX: The report from Gallup that 74% of the world smiled yesterday. The same report found that 49% of people said they learned something interesting yesterday; These are separate claims, each measured a single variable one at a time. -Researchers were not trying to show an association between these single variables, and the report did not claim that the people who learned something interesting were more likely to smile.

Examples of When Causal Claims Are a Mistake

-Do family meals really curb eating disorders? 1) Covariance: Yes, there is an association between family meals and eating disorders. 2) Temporal precedence: Did the family meals increase before the eating disorders decreased? It's not clear. 3) Internal validity: No. We can't rule out third variable explanations without an experiment. -Other possible explanations: A) Perhaps girls from single-parent families are less likely to eat with their families and are vulnerable to eating disorders, whereas girls who live with both parents are not. B) Maybe high achieving girls are too busy to eat with their families and are also more susceptible to eating-disordered behavior. -EX 2: Does Social Media Pressure Cause Teen Anxiety? The relationship sounds causal, but is it? 1) Covariance: Yes. The results showed that teens who felt more pressure to respond immediately to social media were also more anxious. 2) Temporal precedence: No. this was a correlational study, in which both variables were measured at the same time 3) Internal validity: No. It was not an experiment so it did not rule out possible alternative explanations.

Operationalize and Conceptual Variables

-EX 1: A researcher interested in a conceptual variable, "weight gain" in laboratory rats would just weigh them. -EX 2: A researcher interested in the conceptual variable "income" might operationalize this variable by asking each person their total income last year. -In these two cases, the researcher can operationalize the conceptual variable of interest straightforwardly. -Other times, the concepts researchers study are harder to operationalize because they are difficult to see, touch, or feel. -EX 1: Personality traits, states such as "argumentativeness," -EX 2: Behavior, judgments such as "attempted suicide."

Three Claims

-Frequency claims make arguments about the level of a single, measured variable in a group of people. -Association claims argue that two variables are related to each other; can be positive, negative, or zero; are usually supported by correlational studies, in which all variables are measured; when you know how two variables are associated, you can use one to predict the other. -Causal claims state that one variable is responsible for changes in the other variable; to support a causal claim, a study must meet three criteria, covariance, temporal precedence, and internal validity, which is accomplished only by an experimental study.

Positive Association (Positive Correlation)

-High levels of one variable go with high levels of the other variable, and low levels of one variable go with low levels of the other variable -EX: The headline "New study links exercise to higher pay" is an association in which high goes with high and low goes with low

Negative Association (Negative Correlation)

-High levels of one variable go with low levels of the other variable, and vice versa. -EX: The claim "Coffee drinking linked to less depression in women" obtained a negative association. -High rates of coffee go with less depression, and low rates of coffee go with more depression. -The word negative refers only to the slope; it does not mean the association is somehow bad -To avoid this confusion, refer to it as Inverse Association

Independent Variable

-In an experiment, the variable (cause) that is manipulated. -In a multiple regression analysis, a predictor variable used to explain variance in the criterion variable. -To manipulate a variable means to assign participants to be at one level or the other.

Dependent Variable (outcome variable)

-In an experiment, the variable (effect) that is measured. -In a multiple regression analysis, the single outcome, or criterion variable the researchers are most interested in understanding or predicting.

Important Points for Casual Claims

-Includes the words: could, may, seem, suggest, sometimes, potentially -A headline reads "Music lessons may enhance IQ," it would be more tentative, but would still be still considered a causal claim. -The verb 'enhance' makes it a causal claim -Advice is also a causal claim; it implies that if you do X, then Y will happen. -EX: "Best way to deal with jerks? Give them the cold shoulder." "Boost your salary by hitting the gym." -Causal claims are a step above association claims. Because they make a stronger statement, we need to hold them to higher standards.

EXTERNAL VALIDITY OF ASSOCIATION CLAIMS

-Interrogating external validity and an association claim by asking whether it can generalize to other populations, as well as to other contexts, times, or places. -EX: The association between coffee consumption and depression came from a study of women. -Will the association generalize to men? -You can evaluate generalizability to other contexts by asking whether the link between coffee consumption and depression might be generalizable to other forms of caffeine (such as tea or cola). -EX: If a study found a link between exercise and income, you can ask whether the study, conducted on Americans, can generalize to people in Canada, Mexico, or Japan.

Examples of Association Claim Verbs

-Is linked to -Is at higher risk for -Is associated with -Is correlated with -Prefers -Is more/less likely to -May predict -Is tied to -Goes with

Journalists

-Journalists might make a causal claim from a correlational study. -But correlational studies don't establish temporal precedence or internal validity.

Examples of Casual Claims

-Mothers' Friendships Are Good for Babies' Brains -To Appear More Intimidating, Just Tilt Your Head Down -Music lessons enhance IQ -Babysitting may prime brain for parenting -Why sleep deprivation makes you crabby

Internal Validity

-One of three criteria for establishing a causal claim -In a relationship between one variable (A) and another (B), the extent to which A, rather than some other variable (C), is responsible for changes in B -A study's ability to rule out alternative explanations for a causal relationship between two variables. -A is the only thing that changed

Examples of Association Claim

-Speech Delays Could Be Linked to Mobile Devices -Girls More Likely to Be Compulsive Texters -Suffering a Concussion Could Triple the Risk of Suicide -Countries with More Butter Have Happier Citizens -Single people eat fewer vegetables -Angry twitter communities linked to heart deaths

Not All Claims Are Based on Research

-Stories in the popular media that are not based on research, even if they are related to psychology. -Some claims are based on experience, intuition, or authority -EX 1: The Forgotten Mothers and Babies of Zika -EX 2: A Woman Living with Chronic Pain Describes How It Makes Her Seem "Uninterested" -EX 3: Guys Reveal How They Found Their Therapists -These kinds of headlines, while interesting, are not frequency, association, or causal claims, in which a writer summarizes the results of a poll, survey, or other research study. -Such headlines describe a person's experience with a health problem, raise awareness of rare diseases, or share mental health resources. -But they don't say anything about the frequency of a problem or how it might be solved using research based evidence.

Validity

-The appropriateness of a conclusion or decision, and in general, a valid claim is reasonable, accurate, and justifiable. -In psychological research we do not say a claim is simply "valid." -Instead, psychologists specify which of the validities they are applying.

Claim

-The argument a journalist, researcher, or scientist is trying to make -Researchers make claims about theories based on data. -Journalists make claims when they report on studies they read in empirical journals. -Psychologists use systematic observations, or data, to test and refine theories and claims. -EX: A psychologist makes a claim, based on data they have collected, that a certain percentage of teens attempted suicide last year

Statistical Validity

-The extent to which the data support the conclusions -It is important to ask about the strength of the association and its statistical significance (the probability that the results could have been obtained by chance if there really is no relationship) -The extent to which statistical conclusions derived from a study are accurate and reasonable -Improves with multiple estimates. -Researchers conduct studies more than once and then consider the results of all investigations of the same topic, thus combining many estimates is better than using a single one.

External Validity (generalizability)

-The extent to which the results of a study generalize to some larger population (whether the results from this sample of children apply to all U.S schoolchildren), as well as to to other times or situations (whether the results based on this type of music apply to other types of music -An indication of how well the results of a study generalize to, or represent, individuals or contexts besides those in the study itself

Generalizability

-The extent to which the subjects in a study represent the populations they are intended to represent -How well the settings in a study represent other settings or contexts

Covariance

-The extent to which two variables are observe together, is established by the results of a study -Criteria for establishing a causal claim, in a study's results, the proposed causal variable must vary systematically with changes in the proposed outcome variable. -The study's results show that as A changes, B changes; high levels of A go with high levels of B, and low levels of A go with low levels of B.

Operational definitions (operational variables)

-The specific way in which a concept of interest is measured or manipulated as a variable in a study -In order to test their hypotheses with empirical data, researchers need to develop operational definitions, or operational variables -Can include self-report questionnaires, checking records, and obtaining teachers' observations

Random Assignment

-The use of a random method (flipping a coin) to assign participants into different experimental groups so that each participant has an equal probability of being assigned to any group -Increases internal validity because groups are more equal at baseline, its more reasonable to assume that changes between groups are a result of the intervention/treatment

Frequency Claims Headlines

-Thirty-nine Percent of Teens Admit to Texting While Driving -In the U.S., 71% Support Transgender People Serving in the Military -Screen Time for Kids Under 2 More Than Doubles, Study Finds -4 in 10 teens admit to texting while driving -Middle school kids see 2-4 alcohol ads a day

INTERROGATING THE THREE CLAIMS USING THE FOUR BIG VALIDITIES

-To interrogate a frequency claim, ask questions about the study's construct validity (quality of the measurements), external validity (generalizability to a larger population), and statistical validity (the percentage estimate, its confidence interval, and other estimates of the percentage). -To interrogate an association claim and statistical validity address the strength of a relationship, the precision with which it is estimated, and whether it has been replicated in other studies. -To interrogate a causal claim, ask whether the study conducted was an experiment, which is the only way to establish internal validity and temporal precedence; if it was an experiment, further assess internal validity by asking whether the study was designed with any confounds and whether the researchers randomly assigned participants to groups. -Researchers usually cannot achieve all four validities at once in an experiment, their interest is in making causal statements where they may sacrifice external validity to ensure internal validity.

Correlate

-To occur or vary together systematically, as in the case of two variables. -Variables covary, meaning they are related; as one variable changes, the other tends to change, too.

CONSTRUCT VALIDITY OF ASSOCIATION CLAIMS

-To support an association claim, a researcher measures two variables, so you have to assess the construct validity of each variable. -The headline "Study links coffee consumption to lower depression in women," ask how well the researchers measured coffee consumption and how well they measured depression. -The first variable, coffee consumption, could be measured by asking people to document their food and drink intake every day for a period of time. -The second variable, depression, could be measured using a series of questions developed by clinical psychologists that ask about depression symptoms. -In any study, measuring variables is a fundamental strength or weakness and construct validity questions assess how well such measurements were conducted.

Operationalize

-To turn a conceptual definition into a measured variable or manipulated variable -EX 1: A researcher's interest in the construct "coffee consumption" could be operationalized as a structured question in which people tell an interviewer how often they drink coffee. -EX 2: Spending time socializing is a conceptual variable, and how often a person spends an evening alone, socializes with friends, and sees relatives in a typical week are operational variables.

Construct Validity of Frequency Claims

-When you ask how well a study measured or manipulated a variable, you are interrogating the construct validity, be it smiling, smoking, texting, gender identity, food insecurity, or knowing when news is fake. -When evaluating the construct validity of a frequency claim, the question is: How well the researchers measured their variable of interest. -EX: Consider this claim: "39% of teens text while driving." -There are several ways to measure this variable, though some are better than others; You could: A) Ask teenagers to tell you on an online survey how often they engage in texting while they're behind the wheel. B) Stand near an intersection and record the behaviors of teenage drivers. C) Use cell phone records to see if a text was sent at the same time a person was known to be driving. -You would expect the study behind this claim to use an accurate measure of texting among teenagers, and observing behavior is a better way than casually asking, "Have you ever texted while driving?" -To ensure construct validity, researchers must establish that each variable has been measured reliably (meaning the measure yields similar scores on repeated testings) and that different levels of a variable accurately correspond to true differences in

MAKING PREDICTIONS BASED ON ASSOCIATIONS

-With a + or - association, if we know the level of one variable, we can more accurately guess, or predict, the level of the other variable. -The word predict, does not necessarily mean predicting into the future. It means predicting in a mathematical sense, using the association to make our estimates more accurate. -The stronger the relationship between the two variables, the more accurate our prediction will be. -The weaker the relationship between the two variables, the less accurate our prediction will be. -But if two variables are even somewhat correlated, it helps us make better predictions than if we didn't know about this association. -Both + and - associations can help us make predictions, but zero associations cannot. -EX: If we wanted to predict whether or not a child will be obese, we could not do so just by knowing what time they eat dinner because these two variables are not correlated. -With a zero correlation, we cannot predict the level of one variable from the level of the other.

Interrogating Association Claims

1) Construct 2) External 3) Statistical

Interrogating Frequency Claims

1) Construct 2) External, generalizability 3) Statistical

Association claim, ask about 3 validities:

1) Construct- Ask how well the two variables were measured 2) External- Ask whether you can generalize the result to a population 3) Statistical- Can estimate the strength of the association and the precision of this estimate

The Four Big Validities

1) Construct- How well the variables in a study are measured or manipulated. The extent to which the operational variables in a study are a good approximation of the conceptual variables. 2) External- The extent to which the results of a study generalize to some larger population (whether the results from this sample of teenagers apply to all U.S. teens), as well as to other times or situations (whether the results based on coffee apply to other types of caffeine). 3) Statistical- How well the numbers support the claim, how strong the effect is and the precision of the estimate (the confidence interval). Takes into account whether the study has been replicated. 4) Internal- In a relationship between one variable (A) and another (B), the extent to which A, rather than some other variable (C), is responsible for changes in B.

Examples of Claims

1) Internet bloggers- make claims based on personal experience or observation, "The media coverage of congressional candidates has been sexist" 2) Politicians- make claims based on rhetoric, "I am the candidate of change!" 3) Literature scholars- make claims based on textual evidence, "Based on my reading of the text, I argue that the novel Frankenstein reflects a fear of technology"

To move from the simple language of association to the language of causality, a study has to satisfy three criteria.

1) It must establish that the two variables (the causal variable and the outcome variable) are correlated; the relationship cannot be zero. 2) It must show that the causal variable came first and the outcome variable came later. 3) It must establish that no other explanations exist for the relationship. Therefore, when we encounter a causal claim, we must be sure the study can support it.

3 Types of Associations

1) Positive 2) Negative 3) Zero

Interrogating Causal Claims: Three Criteria for Causation:

1)Covariance: The two variables are related. Association claims fulfill this criterion. 2) Temporal Precedence: one variable comes before the other variable in time. Because the research is manipulating one variable and then measuring the other variable, she knows the manipulated variable comes before the outcome variable, which is measured after the manipulation. 3) Internal validity: a study should be able to eliminate alternative explanation. In other words, Variable A is the only thing that changed

Variable name (conceptual variable): Expressing gratitude to romantic partner

1. Operational definition: Researchers asked people in relationships the extent to which they agree with items such as "I tell my partner often that he is the best." 2. Levels of this variable: 7 levels, from 1 (strongly disagree) to 7 (strongly agree) 3. Is the variable measured or manipulated? Measured

Variable name (conceptual variable): Car ownership

1. Operational definition: Researchers asked people to circle "I own a car" or "I do not own a car" on their questionnaire. 2. Levels of this variable: 2 Levels; own a car or not 3. Is the variable measured or manipulated? Measured

Variable name (conceptual variable): Exposure to disinformation

1. Operational definition: Researchers assigned participants to hear false info either one time or two times. 2. Levels of this variable: 2 levels; hearing the false information once or twice. 3. Is the variable measured or manipulated? Manipulated

Variable name (conceptual variable): What time children eat dinner

1. Operational definition: Using a daily food diary, researchers had children write down what time they ate dinner each evening. 2. Levels of this variable: Researchers divided children into two groups: those who ate dinner between 2 P.M. and 8 P.M, and those who ate after 8 P.M. 3. Is the variable measured or manipulated? Measured

Type I Error

A "false positive" result in the statistical inference process, in which researchers conclude that there is an effect in a population, when there really is none.

Type II Error

A "miss" in the statistical inference process, in which researchers conclude that their study has not detected an effect in a population, when there really is one.

Confidence Interval (CI)

A given range indicated by a lower and upper value that is designed to capture the population value for some point estimate (percentage, difference, or correlation)

Point Estimate

A single estimate of some population value (a percentage, correlation, or a difference) based on data from a sample.

Margin of Error of the Estimate

A statistic, based in part on sample size, indicating the probable true value of a percentage estimate in the population.

Variables that can be EITHER manipulated or measured

If childhood music lessons were your variable of interest, you could measure children already taking music lessons, or you could manipulate this variable if you assigned some children to take music lessons and others to take drama lessons.

Correlational Study

Includes two or more variables, in which all of the variables are measured, and can support an association claim

Level (condition)

One of the possible variations, or values, of a variable.