Questionnaire/ survey design
defining the issue
'Who' refers to the participant 'what' is the subject matter e.g. specific brand 'when' - does the researcher mean the last time, last week, last month, last year or ever? 'where' the (subject matter) is (used/of importance)
7 advantages of a survey method
- Large sample sizes and generalisation - detection of small differences - analysis can be based on multiple variables and the data could lend itself to multiple analysis techniques - administration and recording - collection of standardised data - latent factors and relationships - time frames (past, present, future)
Open ended questions or closed ended?
- Unstructured (open-ended) questions - Structured (closed-ended questions --Dichotomous questions --Multiple-choice questions --Questions in scale measurement formats
Ways of improving chance of reaching the response rate
- Use of incentives - reminders - survey image - ease of completion
Cronbach alpha (coefficient alpha) (what does it assess?)
- assesses the quality of the instrument - a low cronbach alpha (α<0.6) indicates that the sample of items performs poorly in capturing the construct which motivated the measure. - once you have got your data, you can assess the quality of the scales: Some items that do not share an equally common core should be eliminated (determined using reliability analysis)
disadvantages of survey method
- development of survey design - data may lack the detail the researcher needs to address the initial research (or be irrelevant if not designed properly) - limited control over how long it can take to collect the required number of responses - non-response error (bias that occurs when the final sample differs from the planned sample) - biased responses or responses reliant on no knowledge - design may limit what analysis is possible
using ordinary words
- should match vocabulary level of participants - most participants do not understand technical marketing words - your language communicates and puts participants in a particular frame of mind as they can answer the questions you pose - unless the language is meaningful to participants, they will be put in a frame of mind that you may not intend, and be answering different questions from those you set.
Using ambiguous words
- words should have a single meaning - a number of words that appear to be unambiguous have different meanings for different people e.g. usually, normally etc. - ambiguous wording can cause response bias -
3 objectives of a questionnaire/survey
1. It must translate information needed into a set of specific questions that participants can and will answer - need to develop questions (e.g. using an existed measurement scale) that participants can and will answer 2. Must uplift, motivate and encourage the participant to become involved, to cooperate and complete the task - must be able to answer - 'What is the participant 3. The questionnaire should minimise response error - error that arises when participants give inaccurate information or when the answers are mis-recorded or misanalysed
the survey development process (9 stages)
1. Research objectives, resources and constraints 2. determine data collection methods 3. determine question response format 4. decide question wording 5. establish flow and layout 6. evaluate the survey 7. pilot and revise 8. final copy 9. implement
resources and constraints (11)
1. Stimuli or tasks 2. Length of survey - how many constraints would you be exploring (driven by theory and past research)? 3. Structure of the questions 4. Sample precision 5. Incidence rate - what percentage of the defined target population has the key characteristics to qualify for being included in the survey? 6. Budget available - the amount of resources available to the researcher 7. Completion time frame 8. quality requirement - how accurate is the derived information 9. completeness of data - how much data and what degree of detail re needed for the defined research problem? 10. Generalisability - at what level of confidence does the researcher want to make inferences about the defined target population from the data results? 11. Precision - what is the acceptable level of error that the data results may have in representing true population parameters?
8 guidelines for choosing question wording
1. define the issue 2. use ordinary words 3. use unambiguous words 4. avoid leading or biasing questions 5. avoid implicit alternatives 6. avoid implicit assumptions 7. avoid generalisations and estimates 8. use positive and negative statements
12 Questionnaire DO'S AND DON'TS
1. ensure questions are without bias 2. make the questions as simple as possible 3. make the questions specific 4. avoid jargon or shorthand 5. steer clear of sophisticated or uncommon words 6. avoid ambiguous words 7. avoid questions with a negative in them 8. avoid hypothetical questions 9. do not use words that could be misheard 10. desensitise questions by using response bands 11. ensure that fixed questions do not overlap 12. allow for 'other' in fixed response questions
What are attitudes?
A persons favourable or unfavourable evaluations, feelings and tendencies towards an object or an idea.
What is piloting?
A pilot study can be defined as a 'small study to test research protocols, data collection instruments, sample recruitment strategies, and other research techniques in preparation for a larger study
Definition of a questionnaire
A set of questions designed to generate the data necessary to accomplish the objectives of the research project; it is a formalised schedule for collecting information from respondents (McDaniel and Gates 2007)
What is a construct? (examples)
A specific type of concept that exists at a higher level of abstraction than everyday concepts e.g. loyalty, intention, satisfaction, attitude and bheaviour
Measurement of attitudes
Aims to identify the direction and intensity of feelings towards a given attitude object.
Interval scale of measurement basic characteristics common examples marketing example descriptive permissible statistics inferential permissable statistics
BC - Differences between objects can be compared; zero point is arbitrary CE - temperature ME - attitudes, opinions, index numbers DPS - Range, mean, standard deviation IPS - product moment correlation, t-tests, ANOVA, regression, factor analysis
Nominal scale of measurement basic characteristics common examples marketing example descriptive permissible statistics inferential permissable statistics
BC - numbers identify and classify objects CE - student registration numbers, numbers on football shirts ME - gender classification of retail outlets DPS - percentages, mode IPS - Chi square, binomial test
Ordinal scale of measurement basic characteristics common examples marketing example descriptive permissible statistics inferential permissable statistics
BC - numbers indicate the relative positions of the objects but not the magnitude of differences between them CE - rankings of the top four teams in the football world cup ME - ranking of service quality delivered by a number of shops. Rank order of favourite TV programmes DPS - percentile, median IPS - rank order correlation, ANOVA
Ratio scale of measurement basic characteristics common examples marketing example descriptive permissible statistics inferential permissable statistics
BC - zero point is fixed; ratio of scale values can be computed CE - length, weight ME - age, income, costs, sales, market shares DPS - geometric mean, harmonic mean IPS - coefficient of variation
3 types of attitudes towards companies and brands
Cognitive - fish is a healthy food Affective - I like to serve fish to my guests Behavioural - How often do you buy fish?
Survey With the assistance of the computer
Complex skip patterns and randomisation of questions
Other concerns in survey research 2
Context effect Acquiescence ▪Tendency to endorse any assertion made in a question, regardless of its content ▪Acquiescence is more common when a question is ▪Difficult to answer ▪When respondents have been encouraged to guess ▪After they have become fatigued ▪During telephone interviews (than during face-to-face interviews) -presumably because people feel more accountable in face to face interviews ▪Efforts to overcome: Identify people by personality types (see Careless responding research which covers this)
other concerns in survey research 1
Context effect Carryover effects-▪Prior items can provide an interpretive framework for the later items -influencing how the participant views what the later item is about Backfire effects -▪Prior items can determine what is seen by the participant as worth saying and/ or redundant (and judgemental anchors) How to overcome?▪Pretesting would have aimed to identify this ▪Rotate question order (if possible) to ensure you offset the effect occur amongst a large portion of the participants for the same questions
In deciding the choice of words, researchers should consult a dictionary and thesaurus and ask the following:
Does the word mean what we intend?▪Does the word mean the same to out target participants? ▪Does the word have any other meanings? ▪If so, does the context make the intended meaning clear? ▪Does the work have more than one pronunciation? ▪Is there any word or similar pronunciations that might be confused with this word? ▪Is a simpler word or phrase suggested that may be more meaningful to our target participants?
Evaluating the survey
Ensure that the survey meets all the information objectives Ensure that the natures of the data to be collected are what the research desires Divide the survey into a number of sections Ensure that the sections and questions are running in a general-to-specific order Check the branching of questions Numbering of questions and survey Check the crowding of questions
use positive and negative statements
Evidence indicates that the response obtained is influenced by the directionality of the statements -whether stated positively or negatively Use some that are positive and some that are negative
Non-attitudes (pseudo non-attitudes)
False negatives▪People who really have an underlying attitude but decline the opportunity to express their opinion▪Inner thoughts -'I'm not going to say I LOVE this product so will just say I don't know of it' E.g., asked for attitude toward a particular brand's sex products ▪May have difficulty in expressing their affect ▪May represent a small number of participants ▪Introduces error in data collected Real non-attitudes True negatives -not a concern May reflect 20-30% of participants answering the attitudinal question
Attitudes (pseudo-attitudes)
False positives ▪Lacking an underlying attitude but expressing an opinion ▪This is a significant concern for marketing research and needs to be avoided ▪Introduces error into data collected Real attitudes True positives
How to avoid pseudo-attitudes amongst survey data? (2)
Filtering participants (Hawkins & Coney, 1981) based on knowledge or relevance of what is being investigated First emphasised by Gallup (1947) and have since been deemed necessary when investigating attitudes ▪Past research has shown that participants would offer opinions on fictitious entities with studies showing that it reduces false reporting of attitudes (Bishop et al., 1983) Evaluate knowledge of what is being examined within the survey ▪Several questions may be used to determine knowledge or relevance of the subject matter to the participant Downside: Krosnicket al. (2002) states that filtering questions may encourage satisficing
How to avoid pseudo-attitudes amongst survey data? (1)
Filtering participants (Hawkins & Coney, 1981) based on knowledge or relevance of what is being investigated Prior question examining knowledge or familiarity Don't know No opinion
Research objectives
Find out what the research problem is, and what information is required.
Multiple-choice questions
Gives the respondent several choices, important that the options are made comprehensive enough to include every possible response. Usually includes an "other" option.
Avoiding leading or biasing questions
Leading question ▪A question that gives the participant a clue as to what answer is desired or leads the participant to answer in a certain way ▪E.g., Do you think that patriotic French people should buy imported cars when that would put French worker out of employment? ▪A better question would be: Do you think that French people should buy imported cars? Yea saying ▪This bias is the result of some participants' tendency to agree with the direction of a leading question
Person-administered survey
Lengthy, complex and varied questions in conversational style
qualitative data research/collectoin
Open ended questions within the questionnaire - allows the respondent to answer in his own words, with no influence by any specific alternatives. - reveal the issues which are most important to the respondent, and this may reveal findings which were not originally anticipated when the survey was initiated. - respondents can 'qualify' their answers or emphasise the strength of their opinions
Avoid;
Ordering items in such a way that necessitates referral back to earlier questions Priming of later responses by earlier ones Preceding knowledge questions with other questions giving the answers
Forced response/choice in survey questions
Participants are forced to express an opinion because a 'no opinion' option is not provided ▪A participant may, in this case, mark the middle scale position ▪If a sufficient portion of the sample do not have opinions on the topic, marking the middle will distort measures of central tendency and variance ▪Accuracy of data may be improved by including a 'no opinion' category The participants are forced to offer a response to a question even if the participant is having trouble equating item xwith stimuli y By filtering out participants for knowledge related to the topic under investigation -they would avoid gathering a forced response that may be a pseudo-attitude Pressing participants into taking a position (reporting an attitude) using forced response is meaningless Those with no attitudes are of no use to predict behaviour
Final copy -> implementation
Personal distribution and collection Personal distribution and postal or email returns Postal or email distribution and postal email returns consider time
eliminate problems by pilot testing
Pilot testing should always be done All aspects of the survey should be tested, including question context, wording, sequence (i.e., to avoid carryover effects etc), question difficulty, instructions and rewards for taking part in the survey The participants in the pilot-test should be similar to those who will be included in the actual survey Begin the survey by using face to face interviews The pilot test should also be conducted online, postal or telephone methods if those are to be used in the actual survey A variety of interviewers should be used in the pilot testing The pilot test sample size should be small, varying from 15 to 30 participants for the initial testing Use protocol analysis (asking participant to think aloud while answering questions) and debriefing to identify problems After each significant revision of the survey, another pilot test should be conducted, using a different sample of participants The responses obtained from the pilot test should be analysed to check the setup of tables and charts The responses obtained from the pilot test should not be aggregated with the responses from the final survey
How to identify pseudo attitudes amongst survey data?
Post experimental interviewing (Rosenberg 1966; McNemaras cited by Vallaincourt, 1973)May reveal what thought processes they were engaging in for the purpose of the research study Use of open ended questions at the end of the study to ask participants to explain their thinking May enable researcher(s) to identify whether the construct under investigation was truly something that the participant was knowledgeable of
To establish flow and layout;
Present all the individual questions and scales previously developed in a logical order Firstly include a brief introductory section and any instructions applicable to the whole survey Include all necessary instructions within each question Order of questions is very important Will you need filler questions?
Avoid generalisation and estimates
Questions should be specific, not general Questions should be worded so that the participant does not make generalisations or compute estimates ▪E.g., 'What is the annual per capita expenditure on clothing in your household?' ▪Participants would first have to determine the annual expenditure on clothing by multiplying the monthly expenditure on clothing by 12 or even the weekly expenditure by 52 ▪A better way to ask -What is the monthly (or weekly) expenditure on clothing in your household? And 'How many members are there in your household?
other concerns in survey research 3
Satisficing Rather than expending the effort to generate optimal answers, respondents may compromise their standards and expend lessenergy. ▪They may be less thoughtful about a questions meaning ▪They may search their memories less comprehensively ▪They may integrate retrieved information carelessly ▪They may select a response imprecisely Differs to optimising which is about the cognitive processes undertaken by participants to ensure they answer questions optimally
when can satisficing occur?
Satisficing is more likely to occur 1.The greater the task difficulty Function of the difficulty of interpreting ▪the meaning of a question and response choices, ▪the difficulty of retrieving and manipulating information in memory, ▪the pace at which an interviewer reads, ▪the occurrence of distracting events, and more.2.The lower the participants ability Ability is presumably greater among respondents adept at ▪Performing complex mental operations, practiced at ▪Thinking about the topic of a question, ▪Equipped with preformulatedjudgments on the issue. 3.The lower the participants motivation to optimize Factors influencing a participants motivation to optimize include ▪Need for cognition ▪The personal importance of the questions topic to the respondent ▪Beliefs about whether the questionnaire will have useful consequences ▪The behaviour of the interviewer, and fatigue
Telephone-administered survey
Short and simple questions in conversational style
Self-administered and online survey
Simple questions and provision of detailed instructions
4 types of data collection methods
Telephone-administered survey Person-administered survey Self-administered and online survey With the assistance of the computer
attitudes may be reported based on
Those stored in memory ▪Attitudes toward a product category may be based on past experiences and usage -e.g., computers Those constructed on the spot ▪Attitudes toward a specific brand may not have heard of -e.g., AntixaComputing™
Purpose of piloting
To determine whether the questions as they are worded will achieve the desired results To determine whether the questions have been placed in the best order To determine whether the questions are understood by all classes of respondent To determine whether additional or specifying questions are needed or whether some questions should be eliminated To determine whether the instructions to interviewers are adequate
Deciding question wording
What constructs are being assessed? - Seek out reliable scales! Don't just make up questions for constructs (e.g. attitude, intention, perceived behavioural control; unless it is absolutely necessary)
dichotomous questions
a directive form of questioning; these questions ask the customer to choose from two or more options
questions must be;
appropriate to the target population - comprehension - recall - unambiguous Sensitive to the respondents - third person - categorical data
pseudo-attitudes
false / in-genuine attitudes
quantitative data research/collection
provides the respondent with an easy method of indicating his answer - he does not have to think about how to articulate his answer. - 'prompts' the respondent so that the respondent has to rely less on memory in answering a question. - responses can be easily classified, making analysis very straightforward - it permits the respondent to specify the answer categories most suitable for their purposes - likert scales - nominal, ordinal, interval or ratio measures
Difference between survey and questionnaire?
survey = quantitative only. questionnaire = can include qualitative data
Determining question response rate
the quality of data is a function of the response rate (i.e. is it a reflecting the sample profile you aimed for?) Consider: budget, access
Avoid implicit alternatives
▪An alternative that is not explicitly expressed ▪E.g., Do you like to fly when traveling short distances?The alternative of driving is only implicit ▪E.g., Do you like to fly when traveling short distances, or would you rather drive? The alternative of driving is explicit
Avoid Implicit Assumptions
▪Assumptions that are not explicitly stated in the question ▪E.g., Are you in favour of a balanced national budget? Implicit are the consequences that will arise as a result of a balanced national budget ▪Are you favour of a national budget if it would result in an increase in personal income tax? A better way to phrase the question
Information sheet should outline
▪Your participation is voluntary ▪You should not feel pressured to answer questions or respond if they do not wish to ▪Your responses will remain anonymous and confidential ▪Any information supplied that makes you identifiable will be removed ▪For an interview or focus group: You may withdraw before, during and up to one month after the interview takes place ▪For an interview or focus group: Participants will be recorded using an audio recorder
