Research Methods Chapter 14

problems with replication attempts

-Contextually sensitive effects -Sample size -Number of replication attempts •Many Labs Project (MLP): 36 replications -> replication rate rose to 85%

Which of the following statements is TRUE of external validity? -A population always refers to the population of the Earth. -The larger the sample, the better. -External validity is based on how the sample is obtained rather than the size of the sample -.Studies must take place outside the laboratory to have external validity.

-External validity is based on how the sample is obtained rather than the size of the sample

Theory-testing mode and real world

-External validity lower priority -To test research question may need create extremely artificial situation

Which of the following claims is most likely to have been tested in generalization mode? -Four out of ten teenagers can't identify fake news when they see it -Reading stressful news makes adults anxious -People who walk faster live longer

-Four out of ten teenagers can't identify fake news when they see it

Generalization Mode and real world

-Need representative sample -Ecological validity -Often builds upon laboratory settings

When a researcher is in theory-testing mode, _____ is (are) a high priority. A. internal validity B. external validity C. biological measures

A. internal validity

A replication study will generally not help to support external validity when it is a A. conceptual replication. B. replication-plus-extension study. C. direct replication.

C. direct replication.

In their examination of 100 studies, the Open Science Collaboration (OSC) found that only 39% of the studies clearly replicated. Many researchers proposed a variety of reasons for why these replication attempts failed. Which of the following is NOT a proposed reason for the failed replications? -The replication context was too different from the original study. -Data and materials from the original study were shared freely online. -Researchers ran additional participants after looking at the data. -Hypotheses were created after collecting the data.

Data and materials from the original study were shared freely online.

what are problems with publication practices?

Journals prefer new findings & significant differences rather than null effects

Sarah criticized your study because it did not use random sampling. Which reply is most appropriate? a. "It's okay, Sarah—I don't need random sampling because I am prioritizing internal validity." b. "It's okay, Sarah—random sampling doesn't matter if you have a big enough sample, and I do." c. "It's okay, Sarah—I don't need random sampling because I mainly care about generalization." d. "It's okay, Sarah—I am only making a frequency claim."

a. "It's okay, Sarah—I don't need random sampling because I am prioritizing internal validity."

Who are the majority or participants in psych research in north America?

college students

Cleo criticized your study because it did not use random assignment. Which reply is most appropriate? a. "It's okay, Cleo—I don't need random assignment because I am prioritizing internal validity." b. "It's okay, Cleo—random assignment doesn't matter if you have a big enough sample, and I do." c. "It's okay, Cleo —I don't need random assignment because I used an independent-groups design." d. "It's okay, Cleo—I am only trying to support an association claim."

d. "It's okay, Cleo—I am only trying to support an association claim."

What claims are in generalization mode?

frequency claims-always association and causal- sometimes

Dr. Reynolds develops a treatment for social anxiety disorder. After he has shown his treatment to be effective in very controlled laboratory studies, he decides to test it in other settings. He collaborates with clinicians who work in a variety of clinical settings across the country. Dr. Reynolds trains the clinicians on his treatment but doesn't supervise them closely. He also allows patients with some common co-occurring disorders, such as depression, to participate in the study. This study is in _________ mode. data-generating generalization theory-testing analysis

generalization

cultural psychology

how cultural contexts shape human thinking, feeling, and behavior.

HARKing

hypothesizing after results are known

file drawer problem

meta-analysis may overestimate true effect size (b/c null effects not published)

direct replication (exact replication)

original study is repeated as closely as possible

Why might replication studies fail?

problems with original study -sample size -HARKing -fail to report variables showing no effect -P-hacking

scientific literature

related studies conducted by different researchers testing similar variables.

Dr. Reynolds developed a treatment for social anxiety disorder. In his initial test of the treatment, he found it was effective in decreasing social anxiety symptoms in adults who only had social anxiety. Because Dr. Reynolds knows people who have social anxiety disorder also often are depressed, he designs a second study to test whether his treatment decreases social anxiety symptoms in two different groups of people -those who have social anxiety disorder only and those who have both social anxiety disorder and depression. This second study would be which of the following? direct replication meta-analysis replication conceptual replication replication-plus-extension

replication-plus-extension

replication-plus-extension

researchers replicate their original study but add variables to test additional questions

p-hacking

run statistical analyses/add/remove participants until results found

meta analysis (journal article)

statistical analysis yields quantitative summary of literature.

experimental realism

studies done in the lab that create situations in which people experience authentic emotions, motivations, behavior

field setting

study in the real world

literature review (journal article)

summary of how body of evidence supports theory

conceptual replication

the conceptual variables in the study are the same, but the operationalization of variables are different

Replicable

the result has been repeated (not just hypothetical)

WERID

western, educated, industrialized, rich and democratic

Generalization mode

when researchers want to generalize findings from sample to population of interest.

theory-testing mode

when testing association or causal claims to investigate whether there is support for a particular theory

Ecological validity

whether a laboratory study generalizes to real-world settings

Improvements to scientific practice

•Larger sample sizes •Report allanalyses and variables -Open science: sharing dataand materials -Preregistration: preregister methods, hypothesis, statistical analyses before data collection •Journals publish replication; accept preregistered •Funding

Horselenberget al. (2003) conducted an experiment based on Kassinand Kieche(1996) in which the participant was falsely accused of striking a computer key that caused the data to be deleted. The earlier study showed 69% of the participants were willing to sign a confession that they had struck the key. The Horselenberget al. study included a change in procedure by which the signed confession included giving up part of the participation fee. This was which type of study? •conceptual replication •direct replication •meta-analysis •confirmation

•conceptual replication