Chapter 6: Conducting Survey and Self-Report Research

Confidentiality vs. Anonymity

*Confidentiality*- research team knows participant's identity, but safeguards that info *Anonymity*- neither the research team or anyone else knows participant's identity

Validity

-*Construct validity*: refers to whether a scale measures the theoretical construct that it purports to measure. -Evaluation of construct validity requires the use of correlations, *correlation coefficient* (r)

Survey Research

-*Highly used* and valuable form of research -Important to *tie survey research to theory* in order to add greater credibility. -When doing this type of research, *review existing surveys* to learn about good designs.

Preparing the final questionnaire

-*Physical layout* (want a lot of "white space"- don't make it too wordy) -Reproduction and materials -Identification of respondents in the questionnaire

Pretesting and Revision

-*Subsample* for the target population for better results administered in same fashion. -Pretest sample should critically analyze *all aspects*: sensitivity of issues, question wording and order, response categories, reliability checks, physical layout, length of time for answering, and instructions. -Make corrections—if several corrections are necessary, pretest again.

Reliability

-*Test-retest* reliability -*Cronbach's α* (internal consistency): are the questions consistent with the ideas? Do they cluster together? A statistical analysis of the questions

What does a descriptive cross-sectional study answer?

-*What:* What is the prevalence of childhood obesity -*Who:* Grade school children -*Where:* in New York City -*When:* The beginning of fall semester 2011

What does an analytical prospective study answer?

-*What:* incidence of diabetes -*Who:* adults -*Where:* in NYC -*When:* the next year -*Why:* social-economic status (SES) Research question: those with lower SES have higher DM rates?

What is the Delphi Technique?

-A method of reaching group consensus. -Group members answer questionnaires in two or more rounds -Each member generates a list of concerns, issues, etc. -A facilitator provides an anonymous summary of previous round; members have the opportunity to revise their response -Several rounds of discussion. Group consensus is reached -Used to circumvent the power of individuals to sway the group, the bandwagon effect of majority opinion, the manipulation of group dynamics, and the unwillingness of individuals to alter publicly stated positions.

Advantages of e-mail surveys

-Can provide an introductory letter -Fast turnaround time -Low cost -Attachments (video or sound) -May allow for reflective data -Convenient for the researcher -Higher response rate depending on the target population (e.g., in-house personnel for a company) -Wide geographic coverage

Advantages of web based surveys

-Can provide an introductory letter -Low cost -Very fast -Can play sound or show video -Can use complex questioning and automatic item response theory to choose successive items -Software makes for ease of compilation, including preexisting information, and for data analysis -Anonymity can be assured, so good for sensitive issues -Open-ended questions may be answered more readily -Unique formatting (color, fonts) -Data directly entered, reducing error in transferring for data entry

What is CATI? What is a benefit? What is it good for?

-Computer Assisted Telephone Interview -Data recording/entry is eliminated; less error -data is directly entered into the system as the interview takes place- less error -good for getting dietary intake data

Drafting the questionnaire

-Considerations for researchers, respondent, interviewer -*Types of questionnaires (open, close, or combination)* -*Types of questions (dichotomous, multiple choice, ranking, etc.)* -First draft of questionnaires

Disadvantages of personal interviews

-Cost ($ and time) -Interviewer bias -Lack of anonymity

What needs to be done during pre-questionnaire planning?

-Define the problem, hypothesis -Determine the info needed to test the hypothesis -*Review previous research* -Develop preliminary questions

Pretest (important!!!)

-Determining the value to questionnaire design -Evaluating the pre-test -Revising the questionnaire, if necessary -You need to pretest the same population you are studying

Types of Questions

-Dichotomous -Multiple Choice -Nominal Scale -Ranking Questions -Rating Questions -Sentence Completion -Open Ended -Branching Questions (skip pattern questions)

Advantages of personal interviews

-Flexible, allows probing/repeat questions -higher response rate -observe verbal and nonverbal behavior -use complex questions

Branching Q's (skip pattern Q's)

-Have you heard of the new menu labeling law? -Yes (please continue) -No (please skip to Question 3)

What needs to be considered when using a mail survey?

-Length (1-2 pages) -Color (doesn't really have an effect) -Ease of completion and return service (provide self-addressed, stamped envelope) -Incentives (dollar in the envelope; enter in a raffle, etc) -Greater return = less opportunity for response bias

Drawbacks of telephone interviews

-Less motivated (see phone call as a hoax) -little control over the situation (they just hang up) -unlisted numbers -cell phone use??

Multiple Choice Questions

-List of answers for respondent to choose from -Education: < HS, HS, >HS -miles walk per day: < 1, 1-5, 6-9, >= 10

Disadvantages of e-mail surveys

-Many respondents may delete because they get "too many e-mails" now -Survey possibly sent to someone else or answered by another person if no check is there -Generalizability may be in question -Need to have an up-to-date list of e-mail addresses -Lack of true anonymity, although there may be confidentiality

Advantages of mail surveys

-Save money and time -No interviewer bias -Anonymous -Completion by the respondents' convenience -Reach a wide geographic region

What does a focus group seek?

-Seeks *interaction* not a question-answer format -Capitalizes on communication among the participants and the researcher

Disadvantages of mail surveys

-Unanswered questions -Low response rate -Inability to use a complex questionnaire format -Do not get it by the deadline

Advantages of telephone interviews

-faster -respondents remain anonymous -frequent callbacks cost little -potential cost saving (compare with face to face interview)

What are the two types of internet surveys?

1) *E-mail*: similar to mail survey.. However, it's a better way to obtain reflective data 2) *Web Based*: Survey instrument is located on the website

3 types of samples

1) Entire sample 2) Random sample (simple, systematic, stratified, cluster) 3) Convenience sample

What 9 elements should be included in a cover letter for a survey?

1) Identification of who is conducting the study 2) The reason the study is being conducted 3) Why it is important for the respondent to complete the survey 4) Assurance that there are *no right or wrong* answers 5) Assurance of *confidentiality* of information given 6) Assurance of *anonymity* of the respondent 7) Length of time it will take to complete 8) *Date of return/completion* 9) Notice of how to obtain results

What are the 4 types of scales

1) Ordinal 2) Interval 3) Ratio 4) Semantic Differential

Flow Plan for conducting a survey (15 major components)

1) Planning the survey 2) Overall design 3) Method of *data collection* 4) Planning *data analysis* 5) Drawing the *sample* 6) Questionnaire *construction* 7) *Pretest questionnaire* 8) Questionnaire revision 9) Administer survey 10) Code preparation 11) *Verification (errors)* 12) *Data entry (double)* 13) Tabulation (frequency) 14) Analysis 15) Recording & reporting

What are the 3 types of questionnaires?

1) Restricted (Closed) 2) Unrestricted (Open) 3) A combination of both

Commandments for Interviewing (7)

1) Small talk (not to enter an interview cold) 2) Practice a lot 3) Remember purpose and be respectful 4) Act naturally, not over-rehearsed 5) Be a good listener 6) Be cordial and appreciative 7) *Don't accept single-syllable answers*

What are the 3 categories of interview structure?

1) Unstructured 2) Semi-structured 3) Structured

What are the two broad categories of survey design?

1) descriptive 2) analytical

3 ways to determine if it's a good questionnaire

1) factor analysis 2) reliability 3) validity

What are the 4 types of bias in survey research?

1) recall bias 2) response order bias 3) response set bias 4) response bias

Response rate of internet surveys

10-15%

How many people should be in a focus group? How long should it be?

4-8 participants ~2 hours

What is a good response rate?

50% and above is really good ~25-30% is okay, but the higher the better

Advantage of focus groups

Advantageous when sampling people who cannot read or write

__________ Questions in order to elicit greatest response

Combine

Nominal Scale

Ethnicity (white, AA, Hispanic), language (English, Spanish)

When using a mail survey, what should be employed?

Follow up letters or telephone calls should be employed (an increase of 20% response rate)

Sentence completion

I like eating fruits because ____________

Combination of both questionnaires

If a closed-ended questionnaire is used, should have at least one open-ended item to allow respondents to express opinion or thought

When can you use a survey that's published?

If it's in the public domain

Where should sensitive Q's and open-ended Q's go?

Near the end

Open Ended Q's

People do not eat enough F&V, what are the main reasons that prevent people from eating more F&V?

What type of survey has the highest response rate?

Personal interviews

Why is pretesting a survey important?

Pretesting is important to make sure your questionnaires are effective and that people understand how to do them so you get optimal results/data

Restricted (Closed) Questionnaires

Provides fixed-alternative questions that can be answered by a simple "yes" or "no" or by checking an appropriate box

What should survey questions address? (4 things)

Questions should address the *what, who, where* and *when* of the survey.

Unrestricted (Open) Questionnaires

Response categories are not specified, and the respondent is allowed to answer in his or her own words

What type of survey has the lowest response rate?

Self-administered questionnaires (e.g. mailed surveys; online surveys) (telephone questionnaires are in the middle)

_________ reliability-check question pairs (i.e., pairs of questions in which one is stated positively and the other negatively).

Separate

Which come first, simpler or more difficult Q's

Simpler questions should be ahead of more difficult ones.

What does a focus group encourage participants to do?

Talk to each other

Dichotomous Questions

The answer comprises two parts, one of which is to be selected by the respondents -yes, no; -male, female -urban, rural

What is response set bias?

answers with same response repeatedly

NEVER use a survey to generate _______________

causation can only say there's an association

What is response order bias?

checks the response comes to mind easily without due consideration

Semi-structured interviews

contains a core of structured questions which can be used by interviewer for in-depth probing you have a certain question you want to ask, but could go in different direction

Analytical survey design

cross-sectional, case control, prospective

Descriptive survey design

cross-sectional, longitudinal, group comparison

Nature of research questions should determine the research ________

design.

Ratio scaling

equal intervals + a true zero; 120 lb vs. 155 lb

Interval scaling

equal units (intervals) of measurement; zero point is arbitrary (e.g. 75 vs. 55 degree Celsius; 0 degree, what does that mean?)

Example of cluster random sample

ex: 5 boroughs, 5 clusters, do a random selection of clusters, then do simple or systematic sampling of people with those clusters; not every group is used; used to save costs

Example of simple random sample

ex: every person gets a number, drop into a big hat, randomly select

Example of systematic random sample

ex: get a list of participants, pick every 10th person

Example of stratified random sample

ex: males in one group, females in another- then random selection; every group is used; more precise than cluster

Personal interviews can be used as an ________ tool to acquire more information

exploratory

What is response bias?

falsifies answers deliberately

Request ___________ needed for subsequent questions first.

information

Vary questions by _____ and ______

length and type

Place questions in a ______ order when possible.

logical

Interview structure: Reliability increases with ______

objectivity

Ordinal scale

poor, fair, good, very good, excellent, outstanding

Don't change format or wording of a survey b/c then you need to ________ it

revalidate

Avoid establishing a response ____

set.

Length of cover letter for survey

should be permissive and short

Why does response rate need to be reported?

so that the reader knows of any potential bias

Mail surveys must be well ______

timed avoid holidays

What should the interviewer do when they get a single-syllable answer?

use follow-up questions to get more out of them. Ex: "How so?"

Unstructured Interviews

used to obtain information that is very personal and/or potentially threatening; susceptible to subjective bias or error open-ended; works well with in-person interviews

Structured interviews

well-defined pattern is followed, similar to a questionnaire you have a list of questions to ask; works best with a large sample size

Convenience sample is what? What is a negative thing about it?

whoever is willing to participate; potential for selection bias

What's the typical response rate of a mail survey?

~25%

Disadvantages of web based surveys

•Generalization may be difficult since it may not reflect population as a whole •Longer surveys generally have lower completion rates •Depending on controls, same person may answer more than once •Some respondents may question the website authenticity and/or fear their answers will be available to anyone on the web •Restricted to Internet users