Research Methods Ch. 5-8
Snowball sampling
A variation on purposive sampling, a biased sampling technique in which participants are asked to recommend acquaintances for the study.
Observer bias
A bias that occurs when observers' expectations influence their interpretation of the participants' behaviors or the outcome of the study.
Quota sampling
A biased sampling technique in which a researcher identifies subsets of the population of interest, sets a target number for each category until the quotas are filled.
Purposive sampling
A biased sampling technique in which only certain kinds of people are included in a sample.
Reactivity
A change in behavior of study participants (such as acting less spontaneously) because they are aware they are being watched.
Observer effect
A change in behavior of study participants in the direction of an observer's expectation. (aka expectancy effect)
Statistical significance
A conclusion that a result from a sample (such as an association or a difference between groups) is so extreme that the sample is unlikely to have come from a population in which there is no association or no difference.
Cronbach's alpha
A correlation-based statistic that measures a scale's internal reliability.
Strength
A description of an association indicating how closely the data points in a scatterplot cluster along a line of best fit drawn in through them.
Stratified random sampling
A form of probability sampling; a random sampling technique in which the researcher identifies particular demographic categories of interest and then randomly selects individuals within each category.
Oversampling
A form of probability sampling; a variation of stratified random sampling in which the researcher intentionally overrepresents one or more groups.
Self-selection
A form of sampling bias that occurs when a sample contains only people who volunteer to participate
Population
A larger group from which a sample is drawn; the group to which a study's conclusions are intended to be applied.
Known-groups paradigm
A method for establishing criterion validity, in which a researcher tests two or more groups, who are known to differ on the variable of interest, to ensure that they score differently on a measure of that variable.
Poll
A method of posing questions to people on the telephone, in personal interviews, on written questionnaires, or via the Internet.
Survey
A method of posing questions to people on the telephone, in personal interviews, on written questionnaires, or via the internet.
Cluster sampling
A probability sampling technique in which clusters of participants within the population of interest are selected at random, followed by data collection from all individuals in each cluster.
Systematic sampling
A probability sampling technique in which the researcher counts off members of a population to achieve a sample, using randomly chosen interval.
Multistage sampling
A probability sampling technique involving at least two stages: a random sample of clusters followed by a random sample of people within the selected clusters
Ordinal scale
A quantitative measurement scale whose levels represent a ranked order, in which it is unclear whether the distances between levels are equal
Which of these samples is more likely to be representative of a population if 100,000?
A randomly selected sample of 100 people
Convenience sampling
Choosing a sample based on those who are easiest to access and readily available, a biased sampling technique.
Representative sample
A sample in which all members of the population of interest are equally likely to be included (usually through some random method) & therefore the results can generalize to the population of interest.
Biased sample
A sample in which some members of the population of interest are systematically left out, and as a consequence, the results from the sample cannot generalize to the population of interest (aka unrepresentative sample)
Census
A set of observations that contains all members of the population of interest.
Response set
A shortcut respondents may use to answer items in a long survey, rather than responding to the content of each item (aka nondifferentiation)
Correlation coefficient r
A single number, ranging from -1.0 to 1.0, that indicates the strength and direction of an association between two variables.
t test
A statistical test used to evaluate the size and significance of the difference between two means.
Describe which sampling techniques allow generalizing from a sample to a population of interest, and which ones do not.
Allow: Simple Stratified Proportionate Cluster, Multistage Not: Convenience sampling Purposive Snowball Self-selected
Mean
An arithmethic average, a measure of central tendency computed from the sum of all the scores in a set of data, divided by the total number of scores.
Bivariate correlation
An association that involves exactly two variables
Convergent validity
An empirical test of the extent to which a measure is associated with other measures of a theoretically similar construct.
Unobtrusive observation
An observation in a study made indirectly, through physical traces of behavior, or made by someone who is hidden or is posing as a bystander.
Acquiescence
Answering "yes" or "strongly agree" to every item in a survey or interview
Why are convenience, purposive, snowball, and quota sampling not examples of representative sampling?
Because they are not random samples so they do not generalize to the population
What is reactivity? What three approaches can researchers take to be sure people do not react to being observed?
Blend in Wait it out: get used to researchers presence Measure the behavior's result (ex: empty liquor bottles in a residential garbage cans indicates how much alcohol is being consumed in community)
Criterion validity
Both address how well a measure relates to a specific outcome. (Does it predict/correlate)
Explain why a variable will usually have only one conceptual definition but can have multiple operational definitions.
Conceptual definition is the researchers definition of the variable in question at a theoretical level. Operational definition represents a researcher's specific decision about how to measure or manipulate the conceptual variable.
Interrogate the construct validity of a study's variables.
Construct validity: a measurement of how well a variable was measured (or manipulated) in a study Measured variables: Variables whose levels are observed and recorded (with no manipulation)
What are four ways of selecting a biased sample of a population of interest? Which subsets are more likely to be selected in each case.
Convenience sampling: who is easiest to access Purposive sampling: Certain kinds of people they want to study Snowball sampling: Participants are asked to recommend few acquaintances Quota sampling: target a number for each category non randomly
Which requires stronger correlations for its evidence: convergent validity or discriminant validity?
Convergent validity
Content validity
Does it capture all parts of a defined construct?
Discriminant validity
Empirical test of the extent to which a measure does not associate strongly with measures of other, theoretically different constructs.
What are some ways to ensure that survey questions are answered accurately?
Ensure anonymity Use weed out items: Include "filler items" (Ex: Interested about racial attitudes, but also ask about politics, gender rolls, and education) Use implicit measures Ask about actions rather than attitudes
Experimenter expectations
Experimenter has certain expectations -> expectations alter experimenter's behavior toward participants -> expected response is more likely shown by participants
When will it be most important for a researcher to use a representative sample?
Frequency claim If you want to generalize beyond your sample, it matters. You can't always confirm this (you don't actually take a full census of an entire population that you've first sampled) Sometimes you can.... Election polling is confirmed (or not) by election results.
Socially desirable responding
Giving answers on a survey that make one look better than one really is.
Faking bad
Giving answers on a survey that make one look worse than one really is.
Internal reliability
In a measure that contains several items, the consistency in a pattern of answers, no matter how a question is phrased.
What types of sampling errors could occur through internet research/polling?
Internet research uses a nonrandom sample Participants are self-selected volunteers Participants know how to use computers Participants have access to computers Participants are Internet savvy (maybe)
Face validity
Is this a plausible measure of the variable? (does it make sense at a gut level)
Explain how carefully prepared questions improve the construct validity of a poll or survey.
It is crucial that each question be clear and straightfoward to answer so it does not confuse respondents or influence their answers.
Explain why a larger sample is not necessarily more representative than a smaller one.
Larger sample = smaller margin of error Statistical term quantifying the degree of error in the study. If 28% of people support a piece of legislation, with a margin of error of 3, then if you did the pool over and over, 95% of the time your result would be between 25-31%.
What are three potential problems related to the wording of survey questions? Can they be avoided?
Leading questions, double-barreled questions, negative wording. Avoided by testing different wording... If the results are the same no matter the wording, it clearly doesn't matter... If results differ, may need to report results differently for each question
Ethics of behavioral observation
Observing public behavior is considered ethical (no expectation of privacy) Researchers don't report on who they watched specifically Videotaping in public is usually okay too Using 1-way mirrors or private videotaping generally requires permission in advance
Describe the differences between ordinal, interval, and ratio scales.
Ordinal allows you to say "first, second, third" Interval has equal distance between level but no true zero so can't say "twice as much" etc. Ratio has an absolute zero
Self-report measure
People answer questions about themselves in a questionnaire or interview.
Fence sitting
Playing it safe by answering in the middle of the scale fir every question in a survey or interview.
Quantitative variable
Quantitative
Ratio scale
Quantitative measurement in which the numerals have equal intervals and the value of zero truly means "nothing"
Interval scale
Quantitative measurement scale that has no "true zero," and in which the numerals represent equal intervals (distances) between levels
Negatively worded question
Question in a survey or poll that contains negatively phrased statements, making its wording complicated or confusing and potentially weakening its construct validity.
Which of the following correlations is the strongest: r = .25, r = -.65, r = -.01, or r = .43?
R= -.65 because it's closest to -1.0
In your own words, describe the difference between random sampling and random assignment.
Random sampling: Researchers draw a sample randomly Random assignment: only used in experiments; researchers randomly assign participants to groups; helps ensure that the groups are the same at the start of the experiment
Physiological measure
Recording biological data.
Observational measure
Recording observable behaviors of physical traces of behaviors.
Describe the kinds of evidence that support the construct validity of a measured variable.
Reliability: how consistent is the measurement? Validity: is it measuring what it's supposed to measure?
Semantic differential format
Response scale whose numbers are anchored with contrasting adjectives.
Rosenthall effect
Rosenthal was one of the researchers for both the intellectual bloomers AND the maze bright/dull studies. Study showed that observers not only see what they expect to see; sometimes they even cause the behavior of those they are observing to conform to their expectations.
Faking good
Same as socially desirable responding.
Name three common ways in which researchers operationalize their variables.
Self report Observational Physiological
For each of the three common types of operationalizations (self-report, observtional, and physiological) indicate which types of reliability would be relevant.
Self-report: test-retest and internal reliability Observational: interrater reliability Physiological: interrater
What are five techniques for selecting a representative sample of a population of interest? Where does randomness enter into each of these five selection processes?
Simple Stratified Proportionate Cluster Multistage During random selection
To establish criterion validity, researchers make sure the scale or measure is correlated with
Some relevant behavior or outcome.
Explain why external validity often matters for a frequency claim.
The findings need to generalize to a larger population or to other settings
Masked design
Study design in which the observers are unaware of the experimental conditions to which participants have been assigned (aka blind design)
Forced-choice format
Survey question format in which respondents give their opinion by picking the best of two or more options.
Open-ended question
Survey question format that allows respondents to answer any way they like.
Likert scale
Survey question format; a rating scale containing multiple response options that are anchored by the terms 'strongly agree, agree, neither agree nor disagree, disagree, and strongly disagree. Likert-type scale does not follow this format exactly.
Reliability is about consistency. Define the three kinds of reliability, noting what kind of consistency each is designed to show.
Test-retest reliability: test is given twice, scores from the two are compared to see if same Interrater reliability: two observers rate some type of behavior and agree with each other Internal reliability: do the different items correlate well with each other
Validity
The appropriateness of a conclusion or decision.
Test-retest reliability
The consistency in results every time a measure is used.
Reliability
The consistency of the results of a measure
Interrater reliability
The degree to which two or more coders or observers give consistent ratings of a set of targets.
Sample
The group of people, animals, or cases used in a study; a subset of the population of interest.
Effect size
The magnitude of a relationship between two or more variables.
Simple random sampling
The most basic form of probability sampling, in which the sample is chosen completely at random from the population of interest.
Probability sampling
The process of drawing a sample from a population of interest in such a way that each member of the population has an equal chance of being included in the sample, usually via random selection.
Observational research
The process of watching people or animals and systematically recording how they behave or what they are doing.
Slope direction
The upward, downward, or neutral slope of the cluster of data points in a scatterplot.
Random assignment
The use of a random method to assign participants into different experimental groups.
What do face validity and content validity have in common?
They both are subjective ways to assess validity.
For which topics, and in what situations, are people most likely to answer accurately to survey questions?
They make an effort to think about each question, they don't worry about looking good or bad, simply because they are unable Self-report is the best way.
What is the difference between observer bias and observer effects? How can such biases be prevented?
Train observers well Create clear rating scales (codebooks) Multiple observers
Double-barreled question
Type of question in a survey or poll that is problematic because it asks two questions in one, thereby weakening its construct validity.
Leading question
Type of question in a survey or poll that is problematic because its wording encourages only one response, thereby weakening its construct validity.
Categorical variable
Variables whose levels are categorical
Why do you think researchers might decide to use an unrepresentative sample, even though a random sample would ensure external validity?
When studying association or causal claims. When it actually matters (reviews on shoes online)
Describe how researchers can make observations with good construct validity.
When they can avoid three problems: observer bias, observer effects, and reactivity