research methods exam 2

Three types of operationalization

self-report observational physiological

Todd is studying the effect of popularity on academic success for his research methods project. He decides to measure popularity by asking each elementary school student to tell him how many friends he or she has. He assumes that more friends means the student is more popular. Which of the following best describes this variable? A ratio scale of measurement A qualitative variable An other-report measure A categorical variable

A

Which of the following is NOT an empirical test of validity? a) Face validity b) Criterion validity c) Divergent validity d) Convergent validity

A

What do you think might be some advantages and disadvantages of using a forced-choice format?

Advantage: Easy to analyze. Prevents fence-sitting (a problem with Likert scales we will address later). Disadvantage: Oversimplifies. Does not distinguish between strong and weak opinions, or degrees of confidence.

What do you think might be some advantages and disadvantages of using open-ended questions on a survey?

Advantage: People can respond however they want. You might learn something unexpected. Disadvantage: Can be difficult or impossible to score/analyze using statistical methods.

various ways to encourage honest responses on surveys.

Anonymous data collection Special "catch" questions Ask people's friends to rate them. Implicit tests

For his research methods class, Felipe plans to watch how students treat other children in their classrooms who have attention-deficit/hyperactivity disorder (ADHD). He will evaluate how positively or negatively the children are treated by their classmates. This is an example of what type of measurement? Self-report measurement Observational measurement Physiological measurement Archival measurement

B

To operationalize driving ability, a researcher counts the number of cones participants knock over on a driving course. What type of operational definition is this? Physiological Observational Self-report Objective

B

Which of the following is true of operational definitions? There is only one operational definition that is possible for each conceptual definition. The specification of operational definitions is one of the creative aspects of the research process. Conceptual definitions are created after operational definitions are determined. Operational definitions and conceptual definitions are the same thing

B

Why are convergent and discriminant validity often evaluated together? Incorrect Response Both terms mean the same thing. Both involve collecting information from a lot of psychological measures of theoretical interest. Both require the use of scatterplots. Both have to be determined by a panel of experts.

B

A physiological measure operationalizes a variable by recording a participants' a) Observable behaviors b) Intrapersonal thoughts c) Biological data d) Questionnaire answers

C

Another word for discriminant validity is ______________ validity. convergent asymmetrical divergent multiple

C

Which of the following is NOT possible? A measure is neither reliable nor valid. A measure is both valid and reliable. A measure is reliable but not valid. A measure is valid but not reliable.

D.

Imagine the following headline "Today's college students are more narcissistic than ever before". How would you question this study's external validity?

How did they sample the college students?

Imagine the following headline "Today's college students are more narcissistic than ever before". How would you question this study's construct validity?

How was narcissism measured (and how was it measured in the past?)

How would you find out if the two operational definitions of happiness measure the same construct?

One method: test if the results from the questionnaire and Ladder of Life are strongly correlated.

What's the difference between an interval and a ratio scale?

an interval scale has no true zero

How generalizable are the results of a study?

external validity

Research that attempts to reach every member of the population.

census

solutions to reactivity

blend in, wait it out, measure the behavior's results (instead of behavior directly)

example of observer effects:

clever hans

Select a random set of clusters from within a population, then sample everyone within those clusters.

cluster sampling

an abstract or theoretical variable

construct

describe the strength of an association, and can also be handy for evaluating reliability.

corelation coeficients (r)

For example: Should we cut funding for Art and Gym? yes? no?

double-barreled question

in this type of validity, your measure is correlated with a relevant outcome: how well the measure predicts a relevant outcome. How well does a test score predict on-the-job performance?

criterion validity

what are three empirical ways to assess validity?

criterion, convergent, discriminant validity

what are two subjective ways to assess validity?

face & content validity

Example: "Which do you prefer, Coke or Pepsi?" what type of question is this?

forced-choice format

Example: "What are your comments on this class?" what type of question is this?

open-ended

how the researcher measures the construct

operational definition

variables representing a rank order. Example: conference standings in sports

ordinal

a sample with a similar composition to the larger population.

representative sample

Only _________ allow us to make valid inferences about the population of interest.

representative samples

A smaller set of people that is supposed to represent the population.

sample

What does it mean to say a sample is biased?

the traits of the sample differ from the traits of the population

ways in which participants don't always give accurate responses to surveys:

using shortcuts (yea-saying) trying to look good self-reporting "more than they can know" false memories

Answering "strongly agree" to every question on a survey to finish more quickly. (acquiescence, or yea-saying) Or, Answering in the middle of the scale every time to avoid expressing controversial opinions (fence-sitting) how to discourage this?

yea-saying (using shortcuts) reverse-coding, eliminate the middle option

Variables whose values are numbers.

quantitative variables

External validity is extremely important for evaluating ________claims.

frequency

If you ask multiple people to observe the same thing, do they reach the same results? which type of reliability?

interrater

"Do you have any problems with your boss?" "How much do you love eating delicious Papa John's pizza?"

leading questions

A correlation-based statistic called _____________ is commonly used to determine internal reliability. Cronbach's alpha kappa a scatterplot Pearson's r

A

a sample whose composition differs from the larger population. This can occur when certain members of the population are more likely to be sampled than others.

biased sample

Select a random set of clusters from within a population, then sample a subset of people within those clusters.

multistage sampling

What is divergent validity?

the idea that measures of unrelated constructs should not be strongly associated

"Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?"

negatively-worded questions

Variables whose values are different categories.

categorical variables

in this type of validity, your measure is less strongly associated with measures of dissimilar constructs Does the measure correlate less strongly with measures of different constructs?

discriminant validity

this type of validity looks like what you want to measure Does it look like a good measure? (often assessed by asking experts)

face validity

In one study, 73% of students surveyed believed they could recall seeing the first plane hit the tower on 9/11, even though no such footage was shown at the time.

false memories

If you ask the same question in different ways, do you get the same answer? what type of reliability?

internal

Equal intervals between values, but a value of zero doesn't mean "nothing". Example: IQ test

interval

Some things to avoid when writing a well-worded question

leading questions, double-barreled questions, negatively-worded questions, question order effects

People do not always know the true reasons for their preferences (Nisbett & Wilson, 1977).

self-reporting more than they can know

How many subcategories of quantitative variables exist? Two Three Four Five

three

is used more often when it's about social/political opinions.

poll

What are the three ethical principles of the Belmont Report?

respect for persons, beneficence, justice

A sample is biased when

the composition of the sample differs from the population of interest.

What would you do if you witnessed someone being mugged in the street? how would people respond to this by using a shortcut?

trying to look good

randomly assigning your participants to different groups/conditions.

random assignment

randomly selecting the participants for your research study.

random sampling

Equal intervals between values, and a value of zero really means "nothing". Example: Reaction times

ratio

When people change their behavior if they know they are being watched.

reactivity

refers to the appropriateness of an operational definition. Does the test measure what it's supposed to?

construct validity

in this type of validity, the measure contains all the parts that your theory says it should contain: Does it include all the important contents of the construct?

content validity

Sometimes a researcher is not overly concerned with external validity. In these cases, they may settle for biased sampling techniques.

convenience sampling, purposive sampling, snowball sampling, quota sampling

In this type of validity, your measure is more strongly associated with measures of similar constructs. Does the measure correlate with other measures of the same construct?

convergent validity

A number which depends on the average correlation among questions which are supposed to measure the same construct. A value of _____ or higher is generally considered good

cronbach's alpha .7

what are some ways of preventing observer bias and observer effects?

observers trained using coded rating scales, multiple observers (interrater reliability), coders observers not aware of the study's hypothesis

intentionally sampling more people from a group with a low population in order to get statistically meaningful results. The final results must be adjusted so the opinions of the oversampled group are weighted to their actual population.

oversampling

In another study, researchers observed the emotional tone and topics of conversation during family dinners. To code emotional tone, researchers rated parents on a scale from 1 (cold/hostile) to 7 (warm/happy). The average was slightly positive (4.2). In addition, kids talked more about how they didn't like the food, and parents talked more about how healthy it was. What type of reliability would be particularly important here?

interrater-reliability

Another way to evaluate criterion validity: check whether the measure picks up on known differences between groups Example: to test a lie detector, we would want to try it on people we know are lying, and people we know are telling the truth.

known-groups paradigm

Example: "I am able to do most things as well as other people" 2 3 4 5 6 7 strongly disagree strongly agree what type of question is this?

likert scale

The entire set of people that a researcher is interested in.

population

Hosea is studying the relationship between caffeine consumption and problem-solving ability. Which of the following is a categorical way to operationalize caffeine consumption? The number of cups of coffee consumed in a day The number of milligrams of caffeine consumed during the study The frequency of buying energy drinks Whether the participant drank a soda in the 24 hours prior to the study

D

What is a leading question?

a question that is framed in such a way as to encouraged a particular response

what are the three types of reliability?

test-retest interrater internal

Many claims are not meant to generalize to the entire human population. Instead, researchers first identify a _____________ (e.g. latino voters), and figure out how to sample it.

population of interest

(aka random sampling): every member of the population has an equal chance to be sampled.

population sampling

Example: Does your child play sports? How often does your child play?

question order effect

What are two things a researcher can do if he/she is concerned about acquiescence (yea-saying) on a survey?

reverse-code some of the questions ask special "catch" questions that nobody should strongly agree with if they're paying attention

ways to get a biased sample:

sampling only those who are convenient to contact, sampling only those who invite themselves (self-selection)

uses a random selection process. Every member of the population of interest could potentially be chosen. Usually not realistic.

simple random sampling

different ways to do population sampling:

simple random sampling, cluster & multistage samling, stratified random sampling, oversampling, systematic sampling

Establishing construct validity would probably be most important for which of the following? A measure of heart rate A measure of the number of times a person eats alone during a month A measure of spirituality A measure of income

spirituality

Select a variety of groups non-randomly, but randomly sample within those groups

stratified random sampling

is used more often when it relates to opinions about consumer products.

survey

methods of posing questions to people.

surveys and polls

using a systematic (but arbitrary) method to sample participants.

systematic sampling

do you get the same results when you test someone twice? Which type of reliability

test-retest

When a researcher watches people or animals and records what they do.

observational research

Two groups of psychoanalysts were shown the same interview of a 26-year-old man discussing his feelings and work experiences. One group was told the man was a patient. The other group was told he was a job applicant. When asked to describe him, the two groups described the man using totally different adjectives: "innovative" vs. "afraid of his own aggressive impulses". example of?

observer bias

when an observer's expectations influence their interpretation of behavior.

observer bias

what are some issues related to construct validity to avoid:

observer bias, observer effects, reactivity

Similar to observer bias. However, instead of just being biased in their interpretations of behavior, the researcher actually changes the behavior of those they are observing.

observer effects

you could test your participants at two different times, and compute the correlation between scores at time 1 and time 2. IQ is supposed to be stable over time, so you would expect a strong, positive correlation. what is this testing? (which reliability?)

test-retest

The popular press often reports on parents acting violently at their children's hockey games. To estimate the true frequency of such behavior, researchers observed 69 boys and girls hockey games in one Canadian city. They observed 64% positive comments, only 4% negative. How would you question this study's external validity? How could one test the convergent validity of this study?

was the one canadian city they studied a typical example? (generalizability) repeat the study using a different measure of violent behavior