Developing a Questionnaire
Example of Guttman scale
(Least extreme) -Are you willing to permit immigrants to live in your country? -Are you willing to permit immigrants to live in your community? -Are you willing to permit immigrants to live in your neighborhood? -Are you willing to permit immigrants to live next door to you? -Would you permit your child to marry an immigrant? (Most extreme)
Advantages of Self-administered surveys
-Completed at specific sites (workplace, schools, hospital_ -Cost effective for large number of surveys -Honest answers to sensitive questions
Cumulative scale
-Demonstrates accumulated characteristic with each item representing an increasing amount of the attribute being measured -Unequal in their weights -example: Risk factors for heart disease are Smoker, DM, hyperlipidemia, FMHX, CAD. when you start to add them together they are logarithmic in their addition
What are the formats of categorical variables
-Dichotomous variables -Ordinal variables -Nominal variables
Wording of questions: check each question for clarity
-Does each question ask what it is intended to ask? -Is the language of each question clear and neutral? -Will members of the study population understand the language? -Is the question sensitive to potential cultural issues related to language?
Two in one (double barreled) questions
-Example: Do you exercise at least 3 times a week and eat a healthy diet ____yes ____no? -Combines two separate questions one for exercise and one for diet. Separate these. -Avoid using "or", "and"
Vagueness
-Example: Do you exercise regularly? "regularly" is not clear. A person who exercises most days of each week might assume that "regularly" means daily and say "no". Another person who exercises once a month may consider that regular. Better to ask: in a typical week how many days do you exercise for at least 30 minutes?
Faulty assumptions
-Example: Do your gums bleed during regular dental cleanings ____yes ____no? -The question assumes that everyone has routine dental cleanings. If "I do not visit the dentist" is not an answer option, a person who does not have dental cleanings is forced to answer no.
Sensitive questions
-Example: Have you ever hit, scratched, bruised, or otherwise physically injured an intimate partner? -This question is unlikely to be answered truthfully if the response should be yes, and it may raise concerns about confidentiality and potential legal requirements for reporting abuse.
Hypothetical questions
-Example: Have you ever thought that you would like to lose 10 or more pounds? -Anyone could have felt this at some point in time, but the question does not clarify whether this is a long-term longing or a thought that crossed the respondent's mind for the first time upon reading the question
Impossible to recall accurately
-Example: How many servings of carrots did you eat most weeks when you were a child? -Adults will not be able to remember this level of detail about their childhood diets
Double negatives
-Example: I did not find this visit with my doctor to be unpleasant? ____Disagree _____neutral _____agree -The wording if this question makes it hard to figure out whether a person who was satisfied with a visit should agree or disagree
Too much detail
-Example: List any prescription medications you have taken for one month or longer in the past 10 years. -Unless the respondent has had very few prescriptions answering this question is impossible without looking up medical records
What is the issue with neglecting the i don't know answer?
-Forces people to choose answer they are not sure about -Hides important information and can lead to systematic inaccuracies in the data -If question was "when was the last time you checked your blood sugar?" force answer prone to information bias = participants default to providing answer they assume researcher wants to hear
Format of the survey
-IT MATTERS -group questions based on the category/topic. -questions should proceed from general to specific (bland to sensitive) -first set of questions should be neutral -sensitive questions later in the survey -demographic questions at the beginning or the end
Wording of questions: responses carefully worded
-Is the choice of responses clear? -For scaled questions, is the rank order -For questions with unranked categories, is the order of possible responses alphabetical or otherwise neutral?
Length of questionnaire
-Long questionnaires less likely to maintain attention and motivation of respondents →Invalid or unreliable responses -Shorter questionnaires are often more valid than longer ones
Disadvantages of questionnaires
-Misunderstanding/misinterpreting questions or response choices -#1 Major Problem: Low response rates
Advantages of questionnaires
-More efficient than interviews -Larger samples in wider geographical distribution in relatively short time -Standardized - everyone has same questions in same way n Provide anonymity (honest and candid responses) -Good for expressing self-observation phenomena: attitudes, values, perceptions
Issues with types of responses
-Numeric responses: question should state exactly how specific the answers should be. example: height in inches to nearest inch, half inch, cm? -Categorical questions: response categories should be listed. "other" category should be included -Ranked questions: how many entries to include on the scale and whether there will be neutral option -Self report surveys: decisions on whether to add a category for not applicable or I don't kno
Rasch Model
-Probability of specific response (right/wrong) modeled as a logistic function of the person and the item -Function of a line representing a continuum -"Easier" items at base and "harder" items at top -Computer Rasch analysis determines order of difficulty of items and locates them along line. It tries to identify multiple constructs. -obtains a threshold
open ended questions
-Probing feelings and opinions [without bias from researcher] -Good if not sure of all possible responses to a question -Difficult to code to different numbers/types of answers -Respondents do not want to take time to answer question -Answer in a short way that is clear to them but nobody else -Good for interviews
Elements of cover letter
-Purpose and importance of study -Indicate why respondents have been chosen -Assure respondents that survey will be anonymous and encourage them to be honest and that they can refuse to answer any question. If it is not anonymous they must be told it isn't. -Suggest how long it should take to complete -Provide a deadline 2-3 weeks -If sponsored...by whom (same if it is a thesis/dissertation) -Thank respondents for cooperation and stress importance of their response -Offer a summary of the report -Sign the letter, include degrees, affiliation
What should be included in the introduction to the survey?
-Purpose of study -How data will be used -How long to complete -Should offer results of the survey if they want it
Closed-ended questions
-Respondents select an answer from provided choices -Easily coded -Does not allow respondents to express their own viewpoints -May introduce bias
Considerations for closed-ended questions
-Responses should be exhaustive -Include all possible responses, otherwise survey answer is full of bias -Include a n/a category, "don't know" -Categories (answer choices) should be mutually exhaustive: Each answer is unique -Blended answers = bias = data is off - Rationale for ordering response choices: Inherent hierarch and Avoid leading respondent to specific choice -Do not use"all that apply" because it creates an interpretation problem and is difficult to code multiple responses
Order of Questions
-Start with easy or at least general questions before moving to more difficult or sensitive questions -Questions should be in an order that flows naturally from one topic to another with similar questions being grouped
Example of guiding question and hypothesis
-The guiding question: Are perceptions of career success different for male versus female physical therapist? -Hypothesis: Men and women will differ with respect to perceptions of career success. OR Men will rate themselves higher in career success compared with women. OR Men will report greater importance of salary and position compared to women
Advantages of Interviews
-Trained interview will record answers thus ensure accuracy of data -Trained interview will record answers and thus ensure completeness of data
Layout and Formatting
-Use readable and large font -Answer sheet should clearly indicate where and how responses should be marked -White space (blank areas) on page are helpful: separates sections, makes pages visually appealing -clear instructions for skips -cover letter and instructions about how to indicate answers: select the one answer that best describes you, Fill in the oval in front of your answer completely using blue or black ink, Circle all options that apply to you, Write your answer in block capital letters as shown in the example
Commercial Research Tools
-Validated tests are available for certain things: use these don't make your own up 1. Beck Depression Inventory 2. General Health Questionnaire 3. Mini-Mental State Examination (MMSE) 4. Health Related quality of life: SF-36 and SF-12
Administration of the survey
-accessible population must be identified -respondents should be give introduction to the survey -send out reminders
Categorical scales
-based on nominal measurement -frequency counts or percentages
Nominal variables
-close ended→categorical -have no built-in order -What is your favorite type of film? action, comedy, documentary, other
Ordinal Variables
-close ended→categorical are ranked based on an inherent order -what is the highest level of education you have completed? less than high school, high school, some college, college degree
Dichotomous variables
-close-ended→categorical have only two response options (like yes/no) -During your lifetime, have you smoked more than 100 cigarettes?
Disadvantages of interview
-cost and time -lack of anonymity: cant get real truth from personal questions
Guttman scale
-cumulative scale -set of statements is presented that reflects increasing intensities of characteristic being measured -everytime you agree you continue on -if agree with 2 then also with one
What are the formats of close-ended questions?
-date and time variables -numeric variables: have a number. continuous. -categorical variables: no internet order or rank. One is not better than the other. red, blue, green. -paired-comparison variables -rank order variables: some type of inherent order, but don't know the distance between variables.
Demographic information
-describes characteristics of respondents to compare with sample population. sample population must be representative of your source population in order to make inferences based on the results you obtain -can either be put at the beginning or the end of the questionnaire
Advantages of interview
-establish rapport with respondent -elicit up front responses to personal questions -in depth analysis -can see peoples facial expressionist preventing the misinterpretation of constructs
What do open ended questions allow participants to do?
-explain their selections and qualify their responses -give multiple answers -provide responses not anticipated by the researcher
What can surveys be intended for?
-generalization -as a description of a particular group (take data and make inferences based off of it)
How to seek validity
-include survey questions or modules that are identical to the ones used in previous research projects -pilot testing (pretest): ask several volunteers from the target population to complete the survey and provide feedback
Continuous scales
-interval or ratio data -ordinal data requires continuous variable to be collapsed into ranks -pain: minimal, moderate, severe
Questionnaire content
-it is often helpful to start with a list of the main categories of questions to be asked, and then to add detail about the specific topics to be covered
Visual analogue scale (VAS)
-line drawn (usually 100mm) with word anchors at either end that represent extremes
Semantic differential
-measure individual feelings about particular object/concept based on continuum extends between two extreme opposites -typically a 7 point scale -not based on agree/disagree concept
Unstructured interview
-no fixed agenda -more qualitative
Preliminary draft of questionnaire
-panel of colleagues review the initial questions because the researcher is typically too close to see flaws -small representative sample of 5-10 individuals from target population do questionnaire to see reliability -talk to pilot group about wording, missing options, etc -monitor the time it takes to complete the survey
Summative scale
-presents total score with all items contributing equal weight to total -Each category has equal weight
Make guiding questions
-questionnaire must measure what it is intended to measure (validity) -Questions are not asked out of interest but because they represent a specific piece of information to address the research questions -Guiding questions are objectives that delineate what researcher is trying to find out -Focus the content of the questionnaire
Issues with Questions: anonymity
-questions can be asked in numerous valid ways. you must divide which is best in order to protect anonymity -each question/component of the instrument is screened for linkage of participants -age (best to worst): age ranges (best for anonymity, harder to analyze), age in years (can identify outliers), what is your birthday? (too specific)
Self reports
-researcher does not directly observe the respondents behavior or attitudes -only records the respondents answers of them -dependent on how well you answer the question. a function of how good is the question
Rank-order questions
-respondent has a series of responses and is asked to rank the responses on ordinal scale -rank candidates in order of preferences from best to worst
Recall bias
-respondents asked to remember past events especially of sensitive nature -potential for bias or inaccuracy -memory of event is not 100% accurate. the more sensitive, the more off the memory becomes. -this is how an individual remembers something.
Form a Hypothesis
-specify the hypothesis with the guiding questions -direct statistical analysis and conclusions
Structured interview
-standardized set of questions -respondents exposed to same question in same order, same choices
Likert scale
-summative scale used to assess attitudes or values -series of statements presented expressing a viewpoint -respondent is asked to select a ranked repose reflecting agreement/disagreement with each one. code each response with a point value -must have a not applicable category
Issues with open ended questions
-they take longer to ask and answer -they may result in irrelevant answers -recoding for statistical analysis is often difficult
What are the two common formats to list choices to closed-ended questions
-using brackets/box: check appropriate response -asking respondents to circle number/letter that appears before the answer
Designing the instrument
-write series of questions addressing each construct from guiding questions -must test the questions to make sure they do measure the construct before putting them on a survey
Standard response rate for most studies
30-60% -limits external validity
Excellent response rate
60-80%
Answers with a poor scale
Example: How many hours a week do you watch TV? ____0 ____1-3 ____4-7____8 or more Even though most people watch more than 1 hour of TV daily, which would put them in the 8 or more response category they may not want to choose an "extreme" answer Their inaccurate responses will lead to false report. Alternatively, these response options may cause respondents to misread the question as how many hours a day they watch TV
Overlapping answer options
Example: In a typical week, how many days do you eat fish? ____0 ____1-3 ____3-5 ____5-7 Participants who eat fish 3 days a week or 5 days a week will not know which response to select -Violates mutually exclusive
Leading answers
Example: What is your impression about the quality of services provided by Butler University? ____Fair ____Good ____Great ____Excellent This question's response options clearly are intended to lead to a positive response as there are no "poor" options 3 positives, and 1 negative
Leading questions
Example: What is your impression of the quality of work done by the dedicated public servants who work at the county health department? This question clearly intends to lead the respondents toward a positive answer and may unintentionally have the opposite effect
Lack of specificity
Example: What is your income? It is not clear if income refers to earnings per hour, week, month, year or whether it refers to pre or post-tax income
Ambiguous meanings
Example: What kind of house do you live in? Without seeing a list of appropriate responses, it is not clear if the answer should be "an apartment", "a rental", "split-level duplex", or "single-family home"
Habituation
Occurs when respondents have given same answer to many questions in a row (agree, agree, agree, etc) that they continue to reply with same response because it has become routine
Undefined abbreviations
Participants may not know that BPH is short for benign prostatic hypertrophy or that is means enlarged prostate
What is the questionnaire outline?
a detailed outline with each item needed to answer the guiding questions. it allows clear conceptualization of the phenomenon being studied.
When an answer is on a continuum (attitude, quality) what must you provide?
a range of responses -3 to 5 choices and an I don't know or unsure option. this covers all your bases
When do you create the questionnaire outline?
after guiding questions and hypothesis are formed
What are surveys often used to describe?
constructs: attitudes, levels of knowledge, values, experience, current practice, characteristics of a certain group
What is a scale used to distinguish
different intensities of a characteristic -created so that a summary score can be obtained from a series of items
Missing answer options
example: what color are your eyes? ___brown ____blue -many possible eye colors are missing
After formatting what should you check for?
grammatical errors, misspellings, missing questions, gaps in logic, unclear instructions, formatting errors -If too long, cut questions
What is survey best for measuring?
information from subjects rather than measuring performance -not good for objective data: you can measure it directly, just measure it -good for subjective data: how well you liked something.
What response rate is too low-cant be published
less than 30%
Cover letter
mailed questionaries must include cover letters that orients respondents to the survey
Starting point of designing a survey
must start with asking what are you trying to measure: this is defining your research question with reference to a target population
How long should the cover letter be?
no longer than one page
What are the two types of questions?
open ended questions and closed ended questions
What are the two types of surveys?
oral interview written questionnaire: a way to collect data
Big words or jargon
participants may not know what an MI is. say heart attack
Branching
question sequence that follows specific answers with more detailed questions -depends on the response to the initial question as to what the next question is -do you perform clinical activities? yes: go to question 2, no: skip to question 3
After data collection method is decided what is the next step?
recruiting members
Interview
researcher asks respondents specific questions face to face then record their answers for later analysis
What can questionaries be designed for?
self reporting or as scripts for interviews
What is a survey?
series of questions given to a group of subjects
Questionnaires
structured surveys that are self administered
What is the most popular questionnaire for descriptive data in health care research?
survey
What is a questionnaire a tool for?
systematically gathering information from study participants
A valid questionare measures
what it was intending to measure in the population being assessed
