M4 Handout 1: Survey Design

Asking for What Respondents Don't Know? Are most of your teachers liberal?

At best respondents are limited to what they infer from what teachers say and do. If you want an answer to this question, you should survey the teachers.

Double Barreled Question? When was the last time you upgraded your computer and printer?

This is a double-barreled question because it asks about two separate topics (computer and printer) at the same time. Respondents who have upgraded one without upgrading the other wouldn't know how to answer. When you are analyzing the data, how will you know which part of the question was answered? "And" or "or" within a question usually makes it double barreled. Try eliminating the less important topic or create two questions.

Response Options Not Exhaustive? What is your current age? a. 10 to 20 b. 21 to 30 c. 31 to 40

This is an example of a question where the response options are not exhaustive because there is no option available for respondents less than 10 years old or more than forty years old.

Loaded Question? Do you approve of the President's oppressive immigration policy?

This question includes an emotive word (i.e., oppressive) which carries a negatively-charged connotation for most respondents. The respondent is asked to answer based on how it would feel to be oppressed rather than on the merits of the president's policy.

What about overly personal or threatening questions?

When ever possible, avoid asking for embarrassing information or information that would cause the person to feel defensive. Only ask such questions when your research objective requires it.

Leading Question? Now that you've seen how you can save time, would you buy our product?

By citing proof the product is good (i.e., it saves time), the questioner has tipped his hand that he wants a "yes" answer. A loaded question asks the respondent to rely on their emotions more than the facts. Loaded questions contain "emotive" words with a positive or negative connotation. For example, while politicians are willing to associate themselves with democracy and liberty, they try to avoid tags like environmentalist and liberal. They know such "charged" terms can create negative reactions in people regardless of the content of the statement.

What about off-topic questions?

Never seek information that isn't essential to your research objective. Avoid being nosey or asking about things you are tangentially interested to know. Your specific research objective dictates the question you must ask, but also dictates what needn't be asked. If in doubt, leave it out.

Questions That Create Response Sets A response set refers to the tendency for a respondent to answer later questions the same way they answered earlier questions. For example, when a respondent notices that they have answered several questions the same way, they assume the next questions could be answered that way too. They speed up, read less carefully, and often miss that their previous answers are no longer suited to the current question. What two techniques can help?

One technique to prevent response sets is to reverse the wording of some of the survey items. For example, "I always look forward to going to work" can be changed to "I always dread going to work." A respondent who enjoys their job would agree with the first statement, but would have to disagree with the new wording, thus "breaking" any response set they had. A second technique to prevent a response set is to intersperse "dummy" questions among real questions. Dummy questions are irrelevant to your research and won't be scored, but they appear to the respondent to be a natural part of the survey. This makes it more difficult for the respondent to recognize/guess what you are measuring.

Response Options Not Mutually Exclusive? What is your current age? a. 10 or less b. 10 to 20 c. 20 to 30 d. 30 or greater

These response options are not mutually exclusive because a person who is 10 years old fits into both of the first two response options (same with respondents aged 20 and 30). Respondents will find this confusing.

Overly Complex Language or Sentence Structure? If you were in a foreign country and did not speak the language but knew someone who did would you? a. Let the person that spoke the language speak for you b. Try to learn the language on your own c. Not try to learn the language or let the person that you knew that spoke the language speak for you.

The question is long and contains multiple parts. First the respondent has to imagine they are in a foreign country, and then imagine that they can't speak the language and then imagine that they have a traveling companion and then that their companion speaks the language. Response option c is also too complex. Notice that the question contains the word "and." "And," "or", or "but" within a question usually make it overly complex. Notice also that the question contains both a negative element (you can't speak the language) and a positive element (your companion can). Positive and negative elements within the same question usually make it overly complex.