FN 263: Week 5 Part 2

Slowest survey method

mail surveys

Costliest surveys

personal interviews, followed by telephone, then mail surveys

Encouraging people to complete the survey

-Consider how the question and answer choice order can encourage people to complete your survey -Don't put two questions together into one question -Avoid leading questions -Might want to ask a question early as to "plant a seed thought" for an upcoming question -Might help the participant answer a later question better

The two major considerations in surveys

1. Encouraging people to complete the survey 2.Understanding how the order of questions can affect your survey results

What are the two types of Pre-tests?

1. Participating pretests 2. Undeclared pretests

Disadvantages of Internet/Intranet (Web Page) Surveys

-Current Internet use is far from universal, and Internet surveys do not reflect the population as a whole. -People can easily quit in the middle of a questionnaire. -They are not as likely to complete a long questionnaire on the Web as they would be if talking with a good interviewer. -If your survey pops up on a web page, you often have no control over who replies -Anyone from A-Z (Antartica to Zanzibar), cruising that web page may answer. -Depending on the software, there is no control over people responding multiple times to bias the results.

How will we use the data?

-Depends on the objective of the study *Depends on the hypotheses of the study -Should compile it into tables, graphs and publish it, right? -Always consider the limitations, possible errors and biases in the study designs, data collection methods, etc... -Most survey/questionnaire problems can be greatly diminished by using randomization techniques, limiting your survey to the target population, and getting a large enough sample.

Consider how the order of questions or the order of answer choices could affect the results of your survey

-Leave sensitive ones at the end -Easy ------>more difficult -Avoid Habituation—try not to have all the questions have the same answer choices—change it up a bit -When people hear the possible multiple choice answers to questions, they tend to pick from the last couple of answers they heard

Question and Answer Biases

-People sometimes give answers they feel will reflect well on them. -Because people like to think of themselves as normal or average, the range of answer choices you give when asking for a quantity or a frequency can affect the results. -Do not have an interviewer ask a respondent's gender. Have the interviewer fill in the answer themselves

Validating a Questionnaire

-Sample is representative—cannot contain people that will be taking the survey/questionnaire -Sample size is large enough—50-100 is usual amount. Large sample sizes will allow cross-validation -Test-retest reliability—time interval between assessments should consider test bias—not too short, not too long. -Validity and Responsiveness—triangulation methods -Longitutional study pre- & post testing is required

Overview of Pre-testing Survey/Questionnaire

-Test with a small number of people. 50-100 -Test on same kinds of people you will include in your study (target population for survey). Do not give the pre-test to people that will be participating in your survey -Have those people answer the questions and provide responses

Question and Answer Biases

-The desire to please translates into a tendency to pick "agreeing" answers on agreement scales. -While logically the percentage that strongly agrees that "X is good" should exactly equal the percentage that strongly disagrees that "X is bad," in the real world, this is unlikely to be true. -Experiments have shown that more people will agree than disagree. -One way to eliminate this problem is to ask half your respondents if they agree that "X is good" and the other half if they agree that "X is bad." You could then reverse the answers given by the second group. -Is extra work, but it may be worth it if it is important to get the most accurate % of people who really agree with something

Using a Validated Survey/Questionnaire

-Validated for the target population that is being studied *Many times, researchers use validated questionnaire/surveys and add questions to them...called a modified validation questionnaire in the research article. -Much easier than designing your own survey. -Takes years to validate one... -Validation of a survey utilizes the following concepts:

Advantages of Internet/Intranet Surveys

-Web page questionnaires can use colors, fonts and other formatting options not possible in most email surveys. -A significant number of people will give more honest answers to questions about sensitive topics, such as drug use or sex, when giving their answers to a computer, instead of to a person or on paper. People give longer answers to open-ended questions on Web page questionnaires than they do on other kinds of self-administered surveys. Some Web survey software, combines the survey answers with pre-existing information

Undeclared Pretest

-When conducting an undeclared pretest, do not tell respondents that it is a pretest. -The survey is given just as you intend to conduct it for real. -This type of pretest allows you to check your choice of analysis and the standardization of your survey. According to Converse and Presser (1986), if researchers have the resources to do more than one pretest, it might be best to use a participatory pretest first, then an undeclared test.

Disadvantages of Scanning/Scanned Questionnaires

-best suited to "check the box" type surveys ans bar codes -have various methods to deal with text responses, but all require additional data entry time -less forgiving (accurate) than a person in reading a poorly marked questionnaire -requires investment in additional hardware to do the actual scanning

Advantages of Scanning/Scanned Questionnaires

-can be the fastest method of data entry for paper questionnaires -more accurate than a person in reading a properly completed questionnaire

Question/Answer Types

-multiple choice -numeric open end -closed-ended with ordered choices -close-ended with unordered response choices -partially closed ended (uses 'other' category) -text open end -rating numerical scales (Likert is most common) *agreement scale (agree strongly to disagree strongly)

Internet/Intranet (Web Page) Surveys

-rapidly gaining popularity -major speed, cost, and flexibility advantages, but also significant sampling limitations

Advantages of Internet/Intranet Surveys

-web page surveys are extremely fast -a questionnaire posted on a popular website can gather several thousand responses within a few hours -many people who will respond to an email invitation to take a web survey will do so the first day, and most will do so within a few days -no cost once the set up has been completed -pictures can be shown -can use complex question skipping logic, randomizations, and other features not possible with paper questionnaires or email surveys -these features can assure better data

Video, Sound, Graphics

A need to get reactions to video, music, or a picture limits your options. You can play a video on a Web page in a computer-direct interview, or in person. You can play music when using these methods or over a telephone, You can show pictures in those first methods and in a mail survey.

Question ans Answer Biases

Always consider the layout of your questionnaire. This is especially important on paper, computer direct and Internet surveys. You want to make it: *Attractive, *Easy to understand *Easy to complete Web Surveys-Use graphics sparingly Show progress in a web survey (85% done) or people might get frustrated and quit (motivation factor here)

Fastest survey methods

Email and Web Page surveys, followed by telephone interviewing.

What is the KISS approach?

Keep It Short and Simple - start with an introduction -have a title of your research -have your IRB consent form ready -Allow a "Don't Know" or "Not Applicable" or open-ended 'other' response to all questions, except to those in which you are certain that all respondents will have a clear answer (In the DK or NA cases, these are wasted answers as far as the researcher is concerned, but are necessary alternatives to avoid frustrated respondents.)

When are people more likely to answer sensitive questions?

People are more likely to answer sensitive questions when interviewed directly by a computer in one form or another.

Scanning/Scanned Questionnaire

Scanning questionnaires is a method of data collection that can be used with paper questionnaires that have been administered in face-to-face interviews; mail surveys or surveys completed by an Interviewer over the telephone.

Data Collection

T-ypes of statistical tests used will depend on your hypotheses/study objectives! -Commonly: *Predictive =___________ *Difference testing between two equal groups =______________ *Difference testing with more than two equal groups *Looking at 'relationships' =___________ *Collecting nominal data =____________ *Collecting ordinal data =_____________ *Testing with unequal groups or small sample #= non-________________ test

Internet Usage

Web page and email surveys offer offer significant advantages, but you may not be able to generalize their results to the population as a whole

Participating pretest

dictate that you tell respondents that the pretest is a practice run; rather than asking the respondents to simply fill out the questionnaire -Involves an interview setting where respondents are asked to explain reactions to question form, wording and order. -Helps determine whether the questionnaire is understandable.

Most cost effective surveys

email and web page surveys for large samples

Who are less likely to respond to emails?

illiterate and less-educated people rarely respond to mail surveys