Psyc207 Chapter 6: Survey and Questionnaires

Question order:

Earlier items can influence the way respondents interpret later items. Suppose the first item on the survey is rather sensitive. That might influence the way respondents answer subsequent questions because they were "primed" by the first item. One way to control for order effects is to make different versions of the survey with questions in different orders (p. 159).

Well-worded questions

are key to good data and contributes to high construct validity of a measure.

In order to interrogate the construct validity of a survey,

we need to look at how well a variable was measured. To do this, we need to determine that the questions are clear and straightforward to answer.

Survey/poll Formats

1. Open-ended questions What do you think of this class? 2.Forced-choice format Do you like this class so far? Yes or No 3.Likert scale This class if very difficult. Strongly disagree 1 2 3 4 5 Strongly agree 4.Semantic differential format Overall quality of the class The worst class, The best class I've ever taken 1 2 4 5 6 7 8 9 I've ever taken

To create good questions avoid:

1.Leading question Do you agree with voter ID laws that will prevent eligible voters from casting their vote? 2.Double-barreled question Do you enjoy swimming and wearing sunscreen? 3.Negatively worded question People who do not drive with a suspended license should never be punished. Disagree 1 2 3 4 5 Agree

Fence sitting:

Another shortcut is to play it safe by choosing the response in the middle of the scale for all items. This is referred to as fence sitting. This might involve providing a neutral response or responding, "I don't know." Fence sitting weakens construct validity when respondents actually have a positive or negative opinion but their responses indicate neutrality. It is difficult to distinguish fence sitters from those who truly feel neutral. One solution is to remove the neutral (center) option so there are an even number of options. Another solution is to use forced choice.

Self-reporting "more than they can know":

Are people capable of reporting accurately about their feelings, thoughts, and behavior? For example, if I ask you why you always buy a particular brand of yogurt, you might not really know why, yet you may tell me something that sounds reasonable.

Double-barreled questions

Asking two questions in one (for example, do you agree that the Second Amendment to the United States Constitution guarantees your individual right to own a gun and that the Second Amendment is just as important as your other constitutional rights? Do you [ ] support, [ ] oppose, or have [ ] no opinion?). These questions have poor construct validity because people may be responding to the first half of the question, the second half, or both. Thus, the item could be measuring the first construct, the second construct, on both. Ask each question separately (p. 157).

Trying to look good:

Most of us like to look good in other people's eyes, so why shouldn't this extend to the way we respond on surveys?

Rating products:

One study found little correspondence between five-star ratings on Amazon and the ratings of the same products by Consumer Reports, an independent product rating firm (De Langhe, Fernbach, & Lichtenstein, 2016). Researchers found that consumers' ratings were, instead, correlated with the cost of the product and the prestige of its brand. Studies like these suggest that people may not always be able to accurately report on the quality of products they buy.

Likert scale:

People are presented with a statement and asked to use a rating scale to reflect their degree of agreement. A Likert scale is anchored by the terms strongly agree, agree, neither agree nor disagree, disagree, and strongly disagree. A Likert-type scale deviates slightly from this format.

Encouraging Accurate Responses

People can give meaningful responses Sometimes people use shortcuts 1. Response sets 2. Acquiescence (yea-saying) 3. Fence sitting

Forced-choice format:

People provide their opinion by choosing the best of two or more options. They are used in political polls (for example, Which of the following candidates would you vote for for president?), for asking for opinions on current issues (for example, Do you think women should have the right to choose whether to have an abortion? Yes or No), or to determine a preference between two choices (for example, Which of these two statements describes you? I like to be the center of attention. It makes me uncomfortable to be the center of attention.).

Socially desirable responding/faking good:

Person responds in a way that is most socially acceptable. Trying to look better than we are (smarter, stronger, thinner, happier, more successful).

Response sets (figure 6.2, p.168)

Researchers want accurate answers from their respondents. What can they do to encourage this? People can give meaningful responses: Can respondents ever provide accurate responses on surveys? Self-reports are sometimes ideal for getting demographic information about age, gender, ethnicity, income, and so on. In other cases, self-report might be your only choice (for example, What were you thinking about when I called your name?). Sometimes people use shortcuts: Sometimes respondents use shortcuts when answering surveys, perhaps to finish them more quickly. One such shortcut is a response set (aka non-differentiation), which is a way of answering a number of questions in the same way (e.g., strongly agree). Response sets weaken construct validity because respondents aren't saying what they really think.

Semantic differential format:

Respondents are asked to rate a target object using a numeric scale anchored by adjectives (for example, rate your professor on level of difficulty on a scale from 1 [show up and pass] to 5 [hardest thing I've ever done]).

Open-ended questions

Respondents may answer any way they like. Pro: They provide researchers with a lot of rich information. Con: Processing the coding and categorizing diverse responses can be difficult and time-consuming. Therefore, researchers sometimes restrict the answers that people provide in one of the following ways.

Leading questions

Sometimes two questions seem to be asking the same thing, but they yield very different responses depending on how they are worded (for example, of the two questions, How fast do you think the car was going when he hit the other car? and How fast do you think the car was going when he smashed into the other car?, which question do you think would lead people to give a higher speed estimate?). If the goal of a survey is to capture respondents' true opinions, then questions should be worded as neutrally as possible.

Negative wording:

The more cognitively difficult a question is for people to answer, the more confusion there will be, which can reduce the construct validity of the item. Using double negatives can make questions more difficult for respondents to process. Consider the question, "Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?" In order to give your opinion about the Holocaust accurately, you must also be able to unpack the double negative of "impossible" and "never. " Negatively worded questions can reduce construct validity by adding to the cognitive load, which may interfere with getting people's true opinions.

Faking bad:

This is less common, but sometimes respondents try to look bad (more aggressive, more deviant, nastier). One way to avoid this is to remind respondents that their responses are anonymous. However, sometimes this results in respondents not taking the survey seriously if they know their names won't be attached to it. Might include special items in the survey to identify socially desirable responders. Can also ask people's friends to rate them. Researchers also can use computerized measures to evaluate implicit attitudes about sensitive topics (for example, the Implicit Association Test [IAT]).

To create good questions:

Try to be straightforward and clear with your questions Use questions that other researchers have used in the past Consider Question order: Earlier items can influence the way respondents interpret later items.

Encouraging Accurate Responses (trying to look good) Figure 6.3: The Accuracy of Self-Reports (p. 163)

Trying to look good 1.Socially desirable responding/faking good 2.Faking bad Self-reporting "more than they can know" Self-reporting memories of events Rating products

Self-reporting memories of events:

What about self-reporting memories of behaviors? For example, when was the last time you took your car to the car wash? What were you wearing on your 21st birthday? Where were you and what were you doing when you got your acceptance letter to this college or university? People's memories for events are not as accurate as they might like to believe. Furthermore, confidence and accuracy are not related. People might feel very confident about what they report, even if they're not accurate.

Survey and poll

both mean the same thing: a method of posing questions to people on the phone, in personal interviews, on written questionnaires, or online.

Acquiescence (yea-saying)

is a type of response set in which people answer positively (yes, strongly agree, agree) to a number of items instead of looking at each item individually. For example, a respondent might answer "5" to every item on Diener's scale of subjective well-being., not because he is a happy person, but rather because he is using a yea-saying shortcut. People apparently have a bias to agree with (say "yes" to) any item—no matter what it states (Krosnick, 1999). Acquiescence can threaten construct validity because instead of measuring the construct of true feelings of well-being, the survey could be measuring the tendency to agree or the lack of motivation to think carefully (p. 160).

Survey

is often used when people are asked about a consumer product

Poll

is used when people are asked about their social or political opinions