Research Methods in Psyc; Chapter 5: Identifying Good Measurements, Chapter 6 - Surveys & Observations: Describing What People Do

Negatively worded questions

- Another way survey items can be unnecessarily complicated - Avoid by -> asking questions two different ways (helps with internal consistency)

fence sitting

Playing it safe by answrimg the moddle or neutral part of the scale. Especially when survey items are confusing, controvercial or they simply dont have an opinion in the matter. One way to elminate this is to remove the neutral answer option.

Internal Reliability

A measure that contains several items, the consistency in a pattern of answers, no matter how a question is phrased. Questions are worded differently but intended to be a measure of the same construct. Thus if a person agrees with the first item on the scale should agree with the second item, as well as the 3rd,4th and 5th.

order effects (sequence effects)

A methodological problem that occurs in experiments utilizing within-subjects designs. This occurs when participants' responses change over the course of the experiment due to fatigue, boredom, or the simple passage of time. This problem can almost always be corrected by the use of counterbalancing.

Ratio Scale

A quantitative scale of measurement in which the numerals have equal intervals and the value of zero truly means "nothing" *if there is a zero, it truly represents nothing A researcher asks questions to measure how many times a person answered correctly, the participant may answer zero questions correctly.

Survey

A study, generally in the form of an interview or questionnaire, on the phone or online, that provides researchers with information about how people think and act. Surveys are often used when people are asked about a certain product. Or general situations.

Response Sets (also known as Differentiation)

A type of shortcut respondents can take when answering survey questions.

Response sets

Also known as nondifferentiation, survey questions. Although response sets do not cause many problems for single standalone item, people might adopt asingle way of answering all the questions especially toward the end of a long questionnaire. People might answer all of them positively, negatively, or neutrally. (Lelkes, Krosnick, Response sets weaken construct validity because these survey respondents are not saying what they really think.

Response

An action or change in behavior that occurs as a result of a stimulus.

6. Which of the following is a way of preventing reactivity? a. Waiting for the participants to become used to the observer. b. Making sure the observers do not know the study's hypotheses. c. Making sure the observer uses a clear code- book. d. Ensuring the observers have good interrater reliability.

Anawer a. Waiting for the participants to become used to the observer.

Convergent Validity

Another form of validity evidence is whether there is a meaningful pattern of similarities and differences: If measures correlates strongly with other measures of the same construct, its showing CONVERGENT VALIDITY. Beck's Measure for Depression is a great convergent validity the BDI and the CES-D ( the Center for Epidemiology Studies Depression scale) Those who scored as not depressed on the BDI also scored as not depressed on the CES-D. *if measuring depression, has to correlate with other measures of depression And it should correlate less strongly to measures of a different constructs; demonstrating discriminant validity. (Or Divergents validity Which is a good things)

Known-Groups Paradigm

Examine whether scores on a measure distinguish between previously identified groups * Researchers see whether scores on the measure can dis- criminate among two or more groups whose behavior is already confirmed

Conceptual Variable (also called conceptual definition or Construct)

Researchers definition of a variable at a theoretical level. Rather their definition of their construct. *Can be hard to define Pages 122-124

Evaluating Reliabiliy

Researchers may use two statistical devices for data analysis: Scatter plots: can show interested agreements or disagreements Correlation Coefficients:"r" the association between one version of a measure and another, between one coder and another. Or between an earlier time or a later time.

forced-choice format

Survey method in which people give their opinion by picking the best of two or more options. This type of survey question format is often used in Polls; like asking which of the three candidates respondents are most likely to voter for. Or asking opinions on current issues or a preference between two choices. The narcissistic personality inventory is an example of a psychology measure that uses forced-choice format. a method of presenting test questions that requires a respondent to select one of several possible answers

Construct Validity

The degree to which a test measures what it claims, or purports, to be measuring. How well a study's variables are manipulated or measured. Crucial piece of any psychological research study for FREQUENCY ASSOCIATION OR CAUSAL CLAIMS The Construct Validity of a study's measured variables is something you will interrogate for any type of claim.

Strength

The relationship is strong when dots are closer to the line; it is weak when dots are spread out. The spread thus corresponds to the strength of a relationship.

2. Which of the following correlation coefficients best describes the pictured scatterplot?

a. r=.78 C. r=.03 b. r= -.95 d. r= 45 Answer: B r = -.95

If an observational study suffers from "observer effect," then the researcher might be measuring ___ instead of ___.

conformed behavior or outcome the actual behavior or outcome

Types of Validity

face, content, criterion, construct, convergent, discriminant,

Ordinal Scale

measurement applies when the numerals of a quantitative variable represent a ranked order * 1st, 2nd, 3rd *intervals may be unequal distances or degrees between 1st second and third. (Winning by an inch or a mile.) *distance from 1st and 2nd can be different from 3rd to 4th

If an observational study suffers from ___, then the researcher might be measuring the "observers' expectations" instead of ___.

observer bias the actual behavior or outcome

Physiological Measure

operationalizes a variable by recording biological data such as brain activity, hormone levels, or heart rate *fMRI *Measuring hormone cortisol released in saliva Pages 124-125

Self-Report Measure

operationalizes a variable by recording people's answers to questions about themselves in a questionnaire or interview *Asking people to self report on frequencies

Socially Desirable Responding

or faking good, the idea is that the reapondents as embarrassed or worried about givkng an unpopular opinion. Thus the respondent wont give an honest amswer or the self report measure. A similar but less common phenomena is faking bad. Researchers can avoid these pitfalls by making sire people know that the surveys are annonymous. By conducting it on line.

fence sitting

playing it safe by answering in the middle of the scale for every question in a survey or interview

open-ended questions

questions that allow respondents to answer however they want, in his or her own words. The drawback to this is that the questions have to be properly coded, which can be difficult and time consuming. Thus researchers must restrict the answered people give.

If an observational study suffers from ___, then the researcher might be measuring the ___ instead of "the typical behavior."

reactivity the best/worst behavior

Discriminant Validity

scores on the measure are not related to other measures that are theoretically different Example even though the BDI should measure the same as other measurements of depression, it should not correlated strongly with measures of constructs other than depression.

Observer bias

tendency of observers to see what they expect to see

Reactivity

How readily a substance combines chemically with other substances.

Double-barreled questions & how they can be avoided

- Asks two questions in one - Avoid by -> asking each question separately

Reactivity & how to minimize

- Change in behavior when study participants know another person is watching - Minimize -> unobtrusive observations, wait it out, & measure the behavior's results

3 potential problems related to the wording of survey questions

- Leading questions - Double-barreled questions - Negatively worded questions

Observer effects & how to minimize

- Observers inadvertently change the behavior of those they are observing - Minimize -> Masked or "blind" research designs, train observers well, multiple observers

Observer bias & how to minimize

- Observers' expectations influence their interpretation of the participants' behavior =s or the outcome of the study - Minimize -> masked or "blind" research designs

2 ways to ensure that survey questions are answered accurately

- Reverse-word some items - Take away the neutral option

Leading questions & how they can be avoided

- Wording leads people to a particular response - Avoid by -> wording every question as neutral as possible

Face Validity

A subjective judgement: of it looks as if it should be a good measure, it has face validity. *how well items seem to be related to the requirements of the job *Head circumference has high face validity as a measure of hat size *Head circumference has low face validity as a measure of intelligence.

Poll

A type of survey particularly geared towards people's social and political opinions. However the terms survey and poll are interchangeable and mean the same thing in terms of scientific research. As both are methods used to pose questions to people on the phone, in personal interview, by questionnaire or online.

2. When people are using an acquiescent response set they are A. Trying to give the researcher the respon they think he or she wants to hear. B. Misrepresenting their views to appear more socially acceptable. C. Giving the same, neutral answer to each question. D. Tending to agree with every item, no matter what it says.

Answer D. Tending to agree with every item, no matter what it says.

In which of the following situations do people most accurately answer survey questions? A. When their answers are anonymous. B. When they are describing the reasons for their own behavior. C. When they are describing what happened a to them, especially after important events. D. When they are describing their subjective experience; how they personally feel abour something.

Answer D. When they are describing their subjective experience; how they personally feel abour something.

Which of the following makes it more likely that behavioral observations will have good interrater reliability? a. A masked study design b. A clear codebook c. Using naive, untrained d. Open-ended responses coders

Answer b. A clear codebook

5. Which one of the following is a means of con- trolling for observer bias? a. Using unobtrusive observations. b. Waiting for the participants to become used to the observer. c. Making sure the observer does not know the study's hypotheses. d. Measuring physical traces of behavior rather than observing behavior directly.

Answer c. Making sure the observer does not know the study's hypotheses.

1. The following item appears on a survey Is your cellphone new, and does it have all the latest features?" What is the biggest problem with this wording? a. It is a leading question. b. It involves negative wording c. It is a double-barreled question d. It is not on a Likert scale

Answer c. It is a double-barreled question

observer effect

Changes in an organism's behavior brought about by an awareness of being observed.

Criterion Validity

Evaluates whether a measure under consideration is related to a concrete outcome, such as behavior, that it should be related to according to the construct being tested. Another form of validity evidence is whether there is a meaningful pattern of similarities and differences. The measure should correlate more strongly with other measures of the same constructs: showing CONVERGENT VALIDITY; And it should correlate less strongly with measures of different constructs- demonstrating DISCRIMINANT VALIDITY. Another way to gather evidence of criterion validity is to use a known groups paradigm. A way in which researchers see whether scores on the measure can discriminate among a set of groups whose behavior is already well understood.

socially desirable responding

Giving answers on a survey (or other self-report measure) that make one look better than one really is. Also called faking good.

faking good

Giving answers on a survey (or other self-report measure) that make one look better than one really is. Also called socially desirable responding.

faking bad

Giving answers on a survey (or other self-report measure) that make one look worse than one really is.

Order Effect

In other words, the order in which questions are asked can also effect the response to a survey. The earlier questions can change the way respondents understand and answer the later questions. The most direct way to control for order effects is to prepare different versions of the survey, with the questions in different sequences. That way researchers, can look for order effects. If the results for the first order effect is different from the results for the second order, researchers can report both sets of results separately. In addition, they might be safe in assuming that people's endorsement of the first question on any survey is unaffected by previous questions.

Scales of Measure

Levels of operational definitions can be coded as: Categorical or Nominal variables: sex, species, or ethnicity Vs. Quantitative or numerical variables: Age, height, or weight.

Interval Scale

Measurements of quantitative variables that meet two conditions: 1st: The numerals represent equal intervals (distances) between levels. 2nd: There is NO TRUE ZERO: A person can get a score of a zero "but the zero does not really mean nothing." *IQ scores have no "zero". The distance between an IQ of a 100-105 represent the same as the distance between IQ scores of 110-115. However a 0 on the an IQ test does not mean a person has no intelligence.

Leading Question

Questions asked in a way that suggest you have a preferred answer. As in priming the respondents answer by wording the question in a leading manner. Speaks on the need to carefully word questions as neutrally as possible and ask the same question worded in different ways to gage the accuracy and validity of the responses.

Reliability

Refers to how consistent the results of a measure are This is the first step then Validity. Both are important. But reliability is not the same as validity There are 3 types of reliability pages 129-134 1) test and retest-ability: the researcher gets consistent scores every time he or she uses the measure. 2) interrater reliability: consistent scores are obtained no matter who does the measurements or observations. 3) internal reliability (also called internal consistency): a study participant gives similar patterns of answers no matter how the researcher has phrased the question.

Validity

Refers to whether the operationalization is measuring what it's supposed to measure or what it is designed to measure. Not the same as reliability. Rather measurement reliability and measurement validity are separate steps in establishing construct validity.

Ways to Measure Variables

Researchers decide hoe to operationalize each variable: They choose among three common types of measures:Naturalistic observational, self-reports, and psychological. And they decide on the most appropriate scale of measurement for each variable they plan to investigate.

When people are most likely to answer accurately to survey questions

Self-report -> gender, identity, socioeconomic status, ethnicity, & so on

Aquiescence

Yeah saying: this happens when people say they strongly agree to every item without really thinking about the question. This threatens construct validity because instead of measuring the construct of true feelings of well being the survey could be measuring the tendency to agree or the lack of motivation to think carefully.

Unobrtusive observation

Study in which researchers sit behind a one way mirror so they can observe how participants interact without letting them know. Thus avoiding observe effect.

Cronbach's Alpha

a correlation-based statistic that measures a scale's internal reliability * The closer it is to 1.0, the better the scale's reliability

Content Validity

To ensure content validity a measure must capture all parts of a defined construct *involves subjective judgement about a measure For example consider the conceptual definition of intelligence, containing elements such as the ability to reason, plan, solve , problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience. To have content validity any operationalization of intelligence should include questions or items to assess each of the these components or dimensions of intelligence.

Correlation Coefficient or "r"

To indicate how close the dots on a scatterplot are to a line drawn through them. a statistical index of the relationship between two things (from -1 to +1) *interrater reliability if r=.7> then has good interrater reliability (don't want a negative correlation) r cant be negative, (close to 1 then you know 2 raters agreed) *strongest closest to 1 Correlation Coefficients:"r" the association between one version of a measure and another, between one coder and another. Or between an earlier time or a later time. The numbers below the scatterplots are the correlation coefficients or "r". The r indicates the same two things as the scatterplots.

Interrater Reliability

Two or more independent observers will come up with consistent or very similar scores, no matter who measures or observes *One person records one data and another person records the same number Most relevant for observational measures

Test- Retest Reliability

a method for determining the reliability of a test by comparing a test taker's scores on the same test taken on separate occasions *Researcher gets consistent scores every time. Test at least twice. * Taking an IQ test and retaking it and getting consistent scores. And repeated again the results again should be consistent.

Observational Measure (sometimes called Behavioral Measure)

a method of measuring a variable by recording observable behaviors or physical traces of behaviors Operationalizes a variable by recording observable behaviors or physical traces of behavior. *example: operationalizes happiness by observing how often a person smiles *Can records physical traces of behavior (counting the number of tooth marks left on a person's pencil)

negatively worded questions

a question in a survey or poll that contains negatively phrased statements, making its wording complicated or confusing and potentially weakening its construct validity

Likert Scale

a scale of containing more than one item usually 1-5 and is anchored by the terms: 1 strongly disagree - 2 disagree - 3 Neither agree or disagree - 4 agree - 5 strongly agree. Strongly Disagree 1-2-3-4-5 Strongly Agree Or vice versa.

masked design

a study design in which the observers are unaware of the experimental conditions to which participants have been assigned

semantic differential format

a survey question format using a response scale whose numbers are anchored with contrasting adjectives Like the Example

Double barreled Question

a type of question in a survey or poll that is problematic because it asks two questions in one, thereby weakening its construct validity. Because the person could be answering her first half of the question, second half or both. Examples

Categorical Variable

a variable that names categories (whether with words or numerals) *sex, species * "can be this, this," but can't be "in between"

4. Classify each result below as an example of face validity, content validity, convergent and discriminant validity, or criterion validity.

a. A professor gives a class of 40 people his five-item measure of conscientiousness (e.g., "I get chores done right away," "I follow a schedule," "I do not make a mess of things"). Average scores are correlated (r=-.20) with how many times each student has been late to class during the semester. Answer Criterion Validity B. A professor gives a class of 40 people his five-item measure of conscientiousness (e.g., "I get chores done right away," "I fol- low a schedule," "I do not make a mess of things"). Average scores are more highly correlated with a self-report measure of tidiness (r = 50) than with a measure of general knowledge (r= .09). Answer Convergent and Discriminant Validity C. The researcher e-mails his five-item measure of conscientiousness (e.g., "I get chores done right away," "I follow a schedule," "I do not make a mess of things") to 20 experts in personality psychology, and asks them if they think his items are a good measure of conscientiousness. Answer Face Validity D. The researcher e-mails his five-item mea- sure of conscientiousness (e.g., "I get chores done right away," "I follow a schedule,' "I do not make a mess of things") to 20 experts in personality psychology, and asks them if they think he has included all the important aspects of consciousness. Answer Content Validity.

3. Classify each of the following results as an example of internal reliability, interrater reliability, or test-retest reliability.

a. A researcher finds that people's scores on a measure of extroversion stay stable over 2 (measured months. Answer Test and Retest b. An infancy researcher wants to measure how long a 3-month-old baby looks at a stimu- Two undergraduates watch a tape of the eye lus on the right and left sides of a screen. each baby looks to the right and to the left. movements of ten infants and time how long The two sets of timings are correlated r= 95. Answer Interrater Reliability c. A researcher asks a sample of 40 people a set of five items that are all capturing how extroverted they are. The Cronbach's alpha for the five items is found to be .65. Internal Reliability

Review Questions 1. Classify each operational variable below as categorical or quantitative. If the variable is quantitative, further classify it as ordinal, interval, or r ratio.

a. Degree of pupil dilation in a person's in a study of romantic couples (measured in millimeters). Answer Quantitive, Ratio b. Number of books a person owns. Answer Quantitative, Ratio c. A book's sales rank on Amazon.com. Answer Quantitative, Ordinal d. Location of a person's hometown (urban, rural, or suburban). Answer Categorical e. Nationality of the participants in a cross- cultural study of Canadian, Ghanaian, and French students. Answer Categorical f. A student's grade in school. Answer Quantitative, Interval.

Quantitative Variable

coded with meaningful numbers *Height , weight, IQ, level of brain activity *continuous There are 3 types Scales of Quantitative Variables: Ordinal, Interval and Ratio Scale.

observational research

gathering primary data by observing relevant people, actions, and situations

Operationalization

the process of turning an concept of interest into a measured or manipulated variable: A variable can be defined in two ways: 1) a Conceptual Variable or conceptual definition or Construct. The researchers definition of the variable in question at a theoretical level. 2) Operational Definition or an Operational Variable. Represents a researchers specific decision about how to measure or manipulate the conceptual variable.

Slope Direction

the upward, downward, or neutral slope of the cluster of data points in a scatterplot *can be positive, negative, or zero