Chapter 7 Research Methods

Example of double-barreled (questions)

"I believe we should stop spending so much money building prisons and put it into building more schools" "Have you ever used meth or know anyone who has ever used meth?"

Example of double-negatives

"I can't stop thinking about a terrorist attack happening" (very rarely...) "Do you disagree that juveniles should not be tried as adults if they commit murder?"

Advantages of electronic surveys

-Access to large sample. -Quick turnaround. -Ability to collect sensitive information. -Availability of an email list of the population. -Interactive and multimedia features can enhance interest in the survey. -Flexible.

Drawbacks of group-administered surveys

-Assembling a group is seldom feasible because it requires a captive audience. -Possibility that respondents will feel coerced to participate and as a result will be less likely to answer questions honestly. -Also, because administering a survey to a group probably requires the approval of the group's supervisor and the survey is likely conducted on the organization's premises, respondents may infer that the researcher is not at all independent of the sponsor and may feel uncomfortable answering some questions.

Benefits of omnibus survey

-Covers a range of topics -Useful for a range of scholars -E.g., GSS, Add Health

Advantages of in-person interviews

-Highest response rates -Can be much longer than other designs; the questionnaire can be complex, with both open-ended and closed-ended questions. -The order in which questions are read and answered can be controlled by the interviewer, and the physical and social circumstances of the interview can be monitored. -Respondents' interpretations of questions can be probed and clarified.

Limitations of omnibus survey

-Limited depth on each topic -Misuse of information

Use multiple questions to create indices or scales

-Limits idiosyncratic variation (single questions may be determined by reactions to words or ideas in question) --The idea is that idiosyncratic variation in response to single questions will average out, so the main influence on the combined measure will be the concept focused on by the questions. -All items must measure same concept (reliability testing)

Avoid confusing phrasing and vagueness

-Same meaning across different types of respondents (no jargon) -Brief and direct -Specific reference period: --Common behaviors or feelings --> one month --Rare behaviors or feelings --> 12 months

Organization and survey design matter

-Title and Cover Letter -Question Order --Context effects --Split-ballot testing (compare different orders)

Floaters

Survey respondents who provide an opinion on a topic in response to a close-ended question that does not include a "don't know" option but will choose "don't know" if it is available.

Fence-sitters

Survey respondents who see themselves as being neutral on an issue and choose a middle (neutral) response that is offered.

Likert-type response

Survey responses in which respondents indicate the extent to which they agree or disagree with statements.

Interpretive question

Questions included in a questionnaire or interview schedule to help explain answers to other important questions.

Demographic questions

Questions on demographic information (sex, age, race, ethnicity) serve as important independent variables for many research studies. Do not include demographic questions that are not necessary for purposes of your research. Used fixed responses.

Contingent question

Questions that are asked of only a subset of survey respondents.

Survey Research

Research in which information is obtained from a sample of individuals through their responses to questions about themselves or others. An efficient method for systematically collecting data from a broad spectrum of individuals and social settings.

Versatility

Researchers can ask respondents questions about almost any topic you can imagine.

Exhaustive responses

Response choices on a survey that allow an option for respondents who require another choice.

Mutually exclusive responses

Response choices on a survey that do not overlap.

Most important concern writing survey questions

Selecting GOOD questions... -To achieve measurement validity -Cannot be unclear or misleading

Computer-interactive voice response (IVR)

Software that uses a touch-tone telephone to interact with people to acquire information from or enter data into a database.

Reliability measures

Special statistics that help researchers decide whether responses are consistent.

Generalizability

Survey methods lend themselves to probability sampling from large populations (and cross populations).

Types of questions

1. Open-ended 2. Closed-ended 3. Combo 4. Interpretive

Beneficial Features of a Survey

1. Versatility 2. Efficiency 3. Generalizability

Survey rules

1. Write clear and meaningful questions. 2. Avoid confusing phrasing and vagueness. 3. Avoid Negatives and Double Negatives. 4. Avoid Double Barreled Questions. 5. Avoid Making Either Disagreement or Agreement Disagreeable. 6. Make Response Choices Mutually Exclusive. 7. Make Response Categories Exhaustive. 8. Utilize Likert-Type Response Categories. 9. Minimize Fence-sitting and Floating. 10. Utilize Filter Questions. 11. Combine Questions into an Index. 12. Demographic questions. 13. Don't Forget to Pretest. 14. Organization Matters. 15. Question Order Matters.

Key guiding principles of questionnaires

1.Maintain Focus -Well-defined (except omnibus) -Each question serves a purpose related to objective 2. Build on Existing Instruments -If evidence indicates it is a good measure 3. Consider Translation

Index

A composite measure based on summing, averaging, or otherwise combining the responses to multiple questions that are intended to measure the same variable sometimes called a scale.

Closed-ended question

A question format in which respondents are provided with explicit responses from which to select. They are easier to analyze, but may obscure what people really think. However, researchers are more likely to get an answer and there is reduced ambiguity.

Double-negative questions

A question or statement that contains two negatives, which can muddy the meaning.

Double-barreled question

A single survey question that actually asks two questions but only allows one answer.

In-person interview

A survey in which an interviewer questions respondents and records their answers.

Phone survey

A survey in which interviewers question respondents over the phone and then record their answers. Best method for relatively short surveys of the general population-response rates tend to be high.

Mailed (self-administered) Surveys

A survey involving a mailed questionnaire to be completed by the respondent.

Filter question

A survey question used to identify a subset of respondents who then are asked other questions (yes, no). -Create skip patterns (no, move to next topic) -If yes, then they get the contingent question

Omnibus survey

A survey that covers a range of topics of interest to different social scientists.

Group-administered survey

A survey that is completed by individual respondents who are assembled in a group.

Electronic survey

A survey that is sent and answered by computer, either through email or on the web.

Computer-assisted self-interviewing (CASI)

A system within which respondents interact with a computer-administered questionnaire by using a mouse and following audio instructions delivered via headphones.

Computer-assisted telephone interview (CATI)

A telephone interview in which a questionnaire is programmed into a computer, along with relevant skip patterns that must be followed. It essentially combines the tasks of interviewing, data entry, and some data cleaning.

Reference period

A time frame in which a survey question asks respondents to place a particular behavior. In general, research shows that the longer the reference period, the greater the underreporting of a given behavior.

Computer-assisted personal interviewing (CAPI)

An interview in which the interviewer carries a laptop computer programmed to display the interview questions and processes the responses that the interviewer types in as well as checking that these responses fall within the allowed ranges.

Difficulties with in-person interviews

Catching all the members of a sample... -Contact rates tend to be lower in city centers, in part because of difficulties in finding people at home and gaining access to high-rise apartments and in part because of interviewer reluctance to visit some areas at night when more people are likely to be home. -Households with young children or elderly adults tend to be easier to contact, whereas single-person households are more difficult to reach.

Efficiency

Data can be collected from many people at relatively low cost and, depending on the survey design, relatively quickly.

Open-ended question

Survey questions to which the respondent replies in his or her own words, either by writing or by talking. Used for exploratory purposes. More detail, draw meaning. Can help determine flaws in other questions.

Drawbacks of electronic surveys

It is not possible to obtain a true representative sample of the U.S. population on the web, since not everyone is connected or has access to the internet. Additionally, households without internet access tend to be older, poorer, and less educated than do those that are connected, so web surveys of the general population can result in seriously biased estimates. Low rates of survey completion: it is easy to stop working on a web survey.

Context effects

Occur in a survey when one or more questions influence how subsequent questions are interpreted.

Mixed-mode survey

Surveys that are conducted by more than one method, allowing the strengths of one survey design to compensate for the weaknesses of another and maximizing the likelihood of securing data from different types of respondents. For example, nonrespondents in a mailed survey may be interviewed in person or over the phone.

Avoid Making Either Disagreement or Agreement Disagreeable

Tendency to agree to avoid seeming disagreeable. You should impartially present both sides of attitude scales in the question itself. Consider replacing the word agree with a range of response alternatives (strongly support to strongly oppose scale) and by defining illegal activity in question: --"To what extent do you support or oppose...vs agree or disagree..." --"Have you ever taken anything from a store without paying for it... vs. Have you ever shoplifted?"

Cover letter

The letter sent with a mailed questionnaire; it explains the survey's purpose and auspices and encourages the respondent to participate. --Sets the tone for the entire questionnaire.

Drawbacks to mailed surveys

The principal drawback in using this method is the difficulty maximizing the response rate-have to rely on people to voluntarily return the surveys. Additionally, there is a hazard of incomplete response-some respondents may skip some questions or just stop answering questions at some point in the questionnaire.

Interview schedule

The survey instrument containing the questions asked by the interviewer for an in-person interview or phone survey.

Questionnaire

The survey instrument containing the questions for a self-administered survey.

Skip patterns

The unique combination of questions created in a survey by filter questions and contingent questions.

Drawbacks of phone surveys

Two matters undermine the validity of a phone survey: --Not reaching the proper sampling units. --Not getting enough complete responses to make the results generalizable. -Because people often are not home, multiple callbacks will be necessary for many sample members. -Interviewers must be prepared for distractions if the respondent is interrupted by other household members. -The growth of telemarketing has led individuals to "just say no" to calls from unknown individuals and organizations or to simply use their answering machines to screen out unwanted calls. -Cell phone users are also harder and more costly to contact in phone surveys. -Households with a cell phone but no landline tend to be younger, so the rate of phone survey participation is declining even more among those 18 to 34 years old.

Split-ballot design

Unique questions or other modifications in a survey administered to randomly selected subsets of the total survey sample so that more questions can be included in the entire survey or so that responses to different question versions can be compared. Allows researchers to determine the effect of question order on responses.

Idiosyncratic variation

Variation in responses to questions that is caused by individuals' reactions to particular words or ideas in the question instead of by variation in the concept that the question is intended to measure.

Pretested

When a questionnaire is taken by a small subsample of respondents to uncover any problems with the questions or response categories.