SOCI211: Questionnaire Design and Construction (7)

Secondary data analysis

Surveys are expensive. Consequently, once the data have been collected it is typically archived so that further work can be done on them. A problem with this is that the issues to which the survey was directed, and the questions used in it, may not coincide precisely with the issues that interest a particular researcher.

Web surveys: disadvantages

Response rates are low. Samples are restricted to the online population and therefore unrepresentative - but this problem is diminishing over time. Respondent privacy will be vulnerable unless the person designing the web site or sending the e-mail messages takes steps to prevent it - by encrypting information and designing a secure web site. The effectiveness of questionnaire design may be compromised by the equipment and software used by the respondent. Nonetheless, there is some evidence of improved performance in recent voting intention surveys.

should 'not sure', 'don't know', or 'no opinion' options be included?

The competing risks are that not including them will force respondents who don't have an opinion on something to express an opinion versus allowing respondents who do have an opinion to opt for one of these 'non attitude' or 'middle' responses. The alternative ways of approaching this issue involve standard format questions, quasi-filter questions, and full filter questions.

Face-to-face interviews: advantages

&B report a high response rate (80% to 85%). There is likely to be a reduced number of "don't knows" and "no answers" - interviewer training can reduce these. Questions can be clarified for respondents. There is the possibility of probing, to clarify responses. They can be used to reach a population without telephone or fixed residence. The interviewer can observer respondents and the setting in which the interview takes place - for example, the quality of accommodation.

data collection methods in surveys

- Self-administered questionnaires - Phone surveys - Web surveys - Face-to-face interviews

respondent recall

- accuracy of recalled information declines over time - questions requireing recall should not be asked

face to face interviews

- creation of a social relationship, encourage cooperation by the interviewer - high response rate - decreased numbers of dont knows - probing for alternative answers - reach a population that has no fixed residence

questions should also avoid

- emotional language -prestige bias - double-barrelled - confusion beliefs with reality - leading questions - exceeding respondents' capabilities - false premises - Soliciting intentions in the distant future - double negatives - unbalanced response categories

Survey research in general: weaknesses

- fail to get at the complexitites of peoples attitudes and behavior - issues addressed by questions do no provide very much info about the broader social context

disadvantages of phone surveys

- limited to people with phones - There is increasing suspicion of phone surveys because they have often been used as a vehicle for telemarketing. - hanging up is easy

Disadvantages of closed-ended questions

- suggest ideas that the respondent would not otherwise have had - espondents with neither opinion nor knowledge of an issue may respond - difficult to tell if a question is being misinterpreted. imple (simplistic?) answers to complex issues.

advantages of closed-ended questions

-Respondent satisfaction (and cooperation), themselves may clarify the meaning of a question - for example, by providing a list of issues the meaning of the word "issue" in the example in the previous slide is made clear to the respondent. - respondents are more likely to answer questions on sensitive issues. -Less articulate/literate respondents are not as disadvantaged. Replication and comparison in future surveys is easier.

disadvantages of open-ended questions

Different kinds of respondent give different degrees of detail in answers (the articulateness/literacy issue). Responses may be irrelevant or contain useless detail. Coding responses is time consuming The range of responses, and forms of responses, may make difficult comparison across respondents. If the questionnaire is administered by an interviewer it may be difficult for the interviewer to write down the answer (accurately).

standard format

Here is a question about another country. Do you agree or disagree with this statement? The Russian leaders are basically trying to get along with America.

quasi-filter

Here is a statement about another country: "The Russian leaders are basically trying to get along with America." Do you agree, disagree, or have no opinion on that?

Questionnaire construction: Questions versus statements

How many hours do you regularly sleep at night?(Less than 4) (5-6)(7-8)(9 or more) How do you feel about the following statement: "I regularly get enough hours of sleep at night" Strongly disagree, disagree, agree, strongly agree If you include both, you can compare the responses using the two different measures.

Disadvantages of self-administered surveys

Low response rate. There may be bias: only people who feel strongly about an issue may be answering. It is not possible to probe in order to clarify respondents answers. Questions cannot be explained. They do not yield information about the physical or psychological condition in which the person filled out the survey.

Advantages of self-administered surveys

Modest cost - printing and mailing expenses, not hiring or training interviewers. People may feel more comfortable giving truthful answers to sensitive questions. The characteristics of an interviewer will not affect the respondent's answer. It allows more time to give well thought-out answers.

web surveys

Their use has increased considerably. They can take one of two forms: i) a respondent is directed to a web site with a questionnaire; ii) the questionnaire is either embedded in, or attached to, an e-mail message. Information on progress through the questionnaire can be provided (percent completed, a bar crossing a box). They have a relatively bad reputation.

Face-to-face interviews: disadvantages

This is a costly and time consuming method. Interviewer characteristics may cause bias. With respect to at least one person, there is no anonymity for the respondent.

phone surveys

This is a very common method of collecting survey data. The interviewer calls the respondent (often at home), asks questions and records the answers. Respondents sampled from lists, phone directories or random digital dialling. Computer assisted telephone interviewing (CATI) can reliably move the interviewer through complex questionnaires involving contingent questions and skips.

advantages of open-ended questions

allow respondents to raise issues that the researcher didn't think of - that is, they make possible unanticipated findings. - respondents can answer in detail and include conditional information. - They may reveal a respondent's logic, thinking process, and frame of reference.

question design

avoid jargon, slang, and abbreviations

purposes of survey research

can be exploratory, descriptive, or explanatory - gather data on a large range of subjects, allowing exploration, description, or testing of explanations

full filter

ere is a statement about another country. Not everyone has an opinion on this. If you do not have an opinion, just say so. Here's the statement: "The Russian leaders are basically trying to get along with America." Do you have an opinion on that? If yes, do you agree or disagree?

web survey: advantages

he cost of data collection is very low. If the web site is set up appropriately the data can be very cheaply extracted and analyzed. People may feel more confident giving truthful answers to sensitive questions.

advantages of phone surveys

he cost tends to be moderate. There are hiring and training costs, but there are no transportation costs. They tend to generate good response rates - 60% or above. The personal characteristics of the interviewer are less likely to be identifiable - and to influence responses. Respondents may be more likely to give socially disapproved answers over the phone than in person (but compare slide 23). If the interviewer is located in a central office their work can be supervised, including the possibility that supervisors can answer interviewer questions. Interviewers' safety isn't compromised.

why large scale?

i) multiple items may be needed to construct indexes and scales; ii) hypotheses may be formulated without sharply defined operationalizations - a survey that contains multiple items allows the exploration of alternative operationalizations of concepts and the development of Guttman scales; iii) once you've contacted a respondent and got his or her agreement to answer questions it makes sense to get as much interesting information from the respondent as possible.

Self-administered Survey (1)

mail survey: a questionnaire, a set of instructions, and envelope for the respondent to send back the completed form. Follow-up mailings increase the survey response and may be administered in various ways: Letters of encouragement to non-respondents; New copies of the questionnaire with a follow-up letter. A total of three mailings is usually about right. Two or three weeks between each mailing is reasonable.

questions should provide categories that are ...

mutually exclusive, exhaustive, balanced

common survey topics

personal characteristivs, how people classify themselves, behaviour, expectations, knowledge of an issue

other examples of surveys

polling firm, market research, project-specific surveys

open-ended

respondent is asked to provide his or her own answer to a question

close-ended

respondent is asked to select an answer from a list provided by the researcher

first step to conducting a survey

sampling

matrix questions

used when you want to ask several questions that have the same set of answer categories - saves space - cons = respondents may provide a response set

Survey research in general: strengths.

useful for describing the characteristics of a large population It can be used to generate large samples. It makes it possible to ask many questions on a given topic It tends to be reliable - standardized questions get developed that can be used across surveys and will tend to generate similar responses. Note, however, that interviewer characteristics and small differences in wording may be a source of response differences.