Chapter 6: Questionnaire Design
Problems with operating with blocks
"Learning effect" on the set of similar questions for different products. Survey respondents acquire knowledge about the survey taker and the dialogue with that survey taker.
semantic differential scale
- 1953 - it measures the attitude towards an object based on the assesment of bipolar attributes in a given scale (1 to 7 or 3 to 3) - difference with the likert scale: it uses expressions of bipolar attributes (not statements) separated by categories without intermediate descriptions. construction stages - generation of adjectives of bipolar sentences without expressing intermediate positions - it is important to randomly mix positive and negative expressions in the left and the right side to avoid automatic answers. - a subsample of judges answer the items. The extreme grading best describes their opinion about each attribute. - selection of the words or expressions that best discriminate (higher correlation or higher differences in means_ - implementation to the final sample of respondents.
Stapel Scale
- 1969 - similar to the likert scale and semantic differential scale - unipolar scale without a neutral point, usually presented vertically - each item has a word or phrase - the scale is expressed with numbers and is set with 10 categories - it is an obligatory scale (even number of categories) - select a number with a + sign for the sentences that, in your opinion, describe accurately the service provided by the university. or select a number with the - sign that do not accurately describe it. - interval scale - it allows the analysis of profiles - it avoids the work of developing sentences or bipolar words
Considering the freedom of choice questions: Open Questions
- Ex. What advantages do you find in the use of the microwave? Advantages: - the respondent is not forced to give a predefined responses (nuances are expressed and answers are richer) - it avoids biases due to false hypothesis of the investigator and discovers new answers or opinions not intuited Disadvantages: - statistical analysis is complex and expensive - answers without interest to the researcher - answers little related to the question -it requires a wide variety of language to express oneself - the survey taker summarizes information when transcribing the answers.
Metric Scales (Quantitative)
- appropriate for quantitative variables (numeric character) and to interpret the results quantitatively - filling out the form is more demanding and difficult with these scales but they extend the possibility of statistical analysis type: - interval scale - ratio scale
Types of measurement scales in Marketing research
- comparative scale (non metric) - non comparative scale (metric)
Structured Questionnaire
- conclusive research (descriptive and causal) - it uses measurement scales that require to provide accurate answers - survey respondents are selected with sampling techniques - representative samples - quantitative analysis - application with static primary resources (survey) and dynamic ones (panel and omnibus surveys)
Guttman Scale
- cumulative scale in which the items (or individuals) are sorted by an attribute, and which the respondents answer positively or negatively (dichotomous) - the answer to an item is affirmative, it is considered to be in accordance with the previous ones - it measures the intensity of attitude hierarchically according to the order of the clauses ex: Now i am going to read 6 sentances, please tell me which one of these describes best the strategic planning activities of your company during the last four years (only indicate one)
Step 4: Definition of information blocks
- definition of homogeneous blocks of information broken down into several questions that allow a coherent structure - to put into groups questions by topic or homogenous blocks to prevent the confusion of the survey respondent - to separate blocks with topic change questions
Likert Scale Advantages
- easy to answer, but the preparation is laborious Regarding thurstone, likert is: - simpler to develop and safer to apply - increased number of possible answers, a possibility that the respondent values the sentences, and reduced number of sentences.
semi-structured survey
- exploratory and conclusive research (descriptive and causal) - it is used to obtain a variety of responses or if they cant be anticipated - it uses measuring scales with accurate answers and leaving some open alternative - respondents are selected with sampling techniques - quantitative analysis - application with static primary resources (survey)
unstructured questionnaire
- exploratory research - it uses guide notes not closed questions - small samples, which are unrepresentative, and untreated statistically - qualitative analysis - application with in depth interviews , focus groups, and face to face surveys at home
key recommendations for the structure design
- first question is very important - simple questions should be at the top ( to increase confidence and interest of respondent) - the order of the questions should not influence the answers (to ask a suggested recall question before a spontaneous recall question) - respondents identification questions should be at the end (at this point a climate of trust has been created and the respondent will better understand the objectives of the study)
considering the purpose of the questions: control questions
it checks the consistency and accuracy of the answers Q4. Do you consumer beer out of meals? Q15: What do you drink out of meals?
Dichotomous Question
it sets out two answer options - used in disjunctive situations - it is often supplemented with a neutral response (do not know / no answer/ or indifferent) Ex: Do you have a car?
Non-metric scales (qualitative or categorical variables)
- to measure qualitative variables (non numeric character) without quantitative significance - simple, but with little chance of analysis type: - nominal - ordinal
Pretest procedures
- tradicional pretest - staging of survey takers - second interviews to the respondents - cognitive interviews - encoding interaction problems during the interview - panel of experts - combination of techniques
Recommendations when formulating questions
- use clear and simple language - questions should be specific and short - questions should be neutral - a questions must not contain two options to value, double questions or repeated negative questions - to exclude difficult issues to answer or involving calculations or estimates - to exclude words with emotional connotations - wording of questions should be inviting and enhance collaboration - questions about sensitive issues require being careful when writing and asking them
how to ask sensitive questions?
-indirect questions -battery of questions -codes -technique of questions with random answers
to exclude difficult issues to answer or involving calculations or complex estimates
WRONG: How many liters of oil did you spend last year? Correct: How many liters of oil did you spend last week? 1 less than 2 2 between 2 & 5 3 more than 5
Reliability
a measure is reliable if it is free of random error, ad yields stable and constant measurements - the repeated measurement should yield the same result - it is related to transitory or situational circumstances (mood, health, tiredness, or being in a hurry)
Questionnaire Structure
a questionnaire is not a list of questions without relationship - it requires a consistent structure defined after designing questions - the order of questions is important because it can effect the answers
A question must not contain two options to value, stay away from double barreled questions or repeated negative questions
Bad Question: How satisfied or dissatisfied are you with the pay and work benefits of your current job? - contains two options to value (pay and work) Bad Question: how useful will this textbook be for students and young professionals in the field? - contains two options to value (students and young professionals)
Considering the purpose of the questions: Filter Questions
Closed questions with few options to try to choose respondent subsamples to continue the questionnaire. It uses instructions on the 'jumps' between questions. It is necessary to control the representation of filtered segments. Ex: what breed of dog do you own? 1. German Shepherd (go to Q7) 2. Other (Go to Q8) 3. None
elements of the structure of the questionnaire
introduction: objectives of study and words to invite respondents to cooperate body: set of questions to ask identifiers: variables on sociodemographic characteristics to define segments
Step 2: Information Needs
- general characteristics of the individual like: demographic / socioeconomic, psychological: lifestyle - mental or internal characteristics of the individual like: personality, motivations, remembered information, attitude, opinion and intentions. - variables of past, present, and future behavior (purchase and consumption)
Nominal scales
- identification of mutually exclusive non numeric categories - equivalence relation y=f(x), where f(x) is a unique connection - order, distance, and arbitrary origin - examples: ID, zip code, category (sex, location, property, status), car registration, football tshirt. - Descriptive statistics: Mode, frequency, contingency coefficient - Inference: non-paranetric test: x^2, Binomial, McNemar
Logic of Thurstone's Model
- if a group of individuals mostly prefer A to B (% of comparison where A is preferred to B ~= 100%) and if only 55% of the group prefers B to C -> differences between values of the scale associated with A and B > difference associated with B and C - to prepare and interval scale of the proportions of these comparisons of stimuli
Constant Sum Scale
- it delivers a constant amount (100 points) among objects or features - unidimensional or multidimensional scale with a priori definition of attributes and direct estimation - it is an ordinal scale although it can provide a measurement of intervals (an object score ranges from 0 to 100) - comparative nature - lack of generalization of the results to other stimuli that are not studied. Ex: please, distribute 100 points among the following set of factors when you are deciding your holiday according to the important: price journey time accommodation category cultural activities tranquility Disadvantages: -difficult calculation when -> there are many or when there are few addends (rounded) - the sum often differs from the maximum ways to avoid possible miscalculation: - removal of answers - reduction of the amounts assigned proportionally
Order (or hierarchical) scale
- it sorts a number of objects according to a criterion (ex. preference) - a prior definition of criteria (attributes) and direct estimation - transformatino of an ordinal scale into an interval scale using Thurstone's theory Types: - unidimensional (several brands for one attribute) - multidimensional (several brands of attributes for one product category) Ex: sort 6 video brands (A,B,C,D,E,F) according to preference. Disadvantages: - little useful with a large number of objects - solution with comparison steps: 1. to classify objects into 3 classes (Upper, middle, lower) 2. to rank objects of each class
Likert Scale
- likert (1932) takes the thurstone scale as starting point - set of positive / negative clauses which judge some aspect of a topic and the degree of agreement with them (1-5 scale) is manifested - varied but not ambiguous statements. - number between 20 and 30 with a balance between positive and negative sentences - applied to a sample and the attitude of each individual is measured by adding the points of their answers - comparison with a standard (average of respondents)
Pairwise (transformation of nominal scale into ordinal scale)
- matrix of pairs (percentage by which the product j is preferred to i), which is converted to an 'earnings' (%>50; value = 1) and 'losses' (%<50; value=0) matrix - the addition of columns is an ordinal scale
Questions should be neurtral
- not to incorporate judgement, assessments or opinions - studies of public opinion require to publish the results and to avoid political uses - non-neutrality derived using inaccurate questions - WRONG: What is your opinion about the speed and friendliness towards the clients in Mercadona? - CORRECT: what do you think of the treatment you receive in Mercadona? Very Quick 1-2-3-4-5 Very slow Very friendly 1-2-3-4-5 very unfriendly
Scales have properties:
- order - distance - origin of numbers - statistical analysis depends on the scale
Ordinal Scales
- order relationship between categories - relationship y=f(x), where f(x) is an increasing function - distance and arbitrary origin - ex: order by preference, difficulty, maturity, level of education, level of agreement with a situation quality, satisfaction - descriptive statistics: median, quantile, spearman rank, correlation - inference statistics: non-parametric tests: Kendall, Friedman, Mann-Whitney, Wald-Wolfowitz
Interval Scale
- ordinal scale fixed unit of measurement (difference between constant intervals) - relationship y=ax+b where b>0 - origin (zero point) and arbitrary measurement unit - the ratio of two intervals is independent of the measuring unit and the zero. - ex: temperature, energy, dates, index number, attitude to product, establishment opinion, likert, thurstone, and stapel scales, association, and semantic differential scale - descriptive statistics: common parametric statistics, arithmetic standards deviation, pearson correlation - inference statistics: parametric tests: t, f regression, anova, factor analysis
Participants
- people responsible for the survey - survey takers - respondents with similar characteristics to population
Measurement Scale assesment
- perfect measurement (without errors) when it captures perfectly a characteristic - obtained value = true value
Factors that explain the difference
- personal and transitory features of the interviewee (mood, health, tiredness or being in a hury) - situational facotrs (changes in the decision making context) (ex. individual or group questions to a couple) - data gathering factors (interviewers way of formulating questions) - instrument of measurement (ambiguity and difficulty of questions) - mechanical factors ( lack of space to register the response)
Questions should be specific and short
- precision - problems with adverbs, which create inaccuracies and confusion - WRONG: Do you think you smoke too much? Yes/No - CORRECT: How many cigarettes do you smoke per day?
Multiple scales measurement
- they consist of multiple related items and selected in order to measure a multidimensional feature - widely used to measure attitudes - these scales generate a score range Not suitable to: - compare the attitudes of different groups of individuals - check the temporary change in the attitudes of the group - more use of attitude questions in Market research than behavior questions - attitude is a precursor of behavior. it is easy to measure
Continuous comparative scale
- to chose a position on a line connecting two ends of a feature to compare two objects - unidimensional scale with a priori definition of the attribute and direct estimation ex: the car with brand A with respect to B is: identical ---X-----------different
Thurston Scale
- proposed by thurstone in 1931 (one of the first) - scale construction under the hypothesis of apparently equal intervals a) preparation of the sentences - preparing a large number of sentences (150) - clear/accurate sentences, consistent with the attitude to study, short and one logical sentence - number of positive, negative, and neutral sentences in the same proportion b) evaluation of the sentences by judges representative of the population (1-11, 1-9, 1-7 scale, neutral term) to what extent the sentences suits the attitude. - does not arise if they agree or disagree with the statements. - extreme categories reflect extreme attitudes. c) selection of sentences: - with semi-interquartile range or discriminant dispersion - R=(Q3-Q1)/2, from the 3rd and 1st quartile (it measures the ambiguity of the sentences) of the judges scores, under 2 - with IQR less than 3 -> greater consistency among judges towards sentence. -selecting a similar number of sentences for each scalar value d) construction of the final scale - sentences are presented to a representative sample of the population, but turned into 2-option closed questions (agree or disagree) e) rating sample individuals - assignment of 1 point to an individuals answer in each sentence he/she agrees and of 0 if she/he disagrees. - to multiply the answer (1 or 0) of each sentence by its median) - score of each individual = ratio of the sum of such products for all sentences and the sum of the ones corresponding to the 'yes' of all sentences.
Questionnaire composition
- questions - instructions for survey respondent selection and approach - annex (map, figures, photographs, cards) - Gifts or incentives
Considering the Subject
- questions about facts and behaviors - action questions - information or knowledge questions - intention questions - conduct questions - opinion questions - attitudinal questions - questions about aspirations or expectations - questions about reasons - memory questions - questions about sensitive issues - questions about identification or sociodemographic characteristics
Ratio scales
- real zero point in origin (not arbitrary) - y=bx, where b>0 - the proportion of one point to another scale is independent of the unit of measurement (absolute magnitudes are compared) - ex: weight, height, length, tone, volume, density, sales, purchases, costs, age, market share - descriptive statistics: harmonic mean, geometric mean, coefficient of variation
association scale
- respondents associate attributes with objects it provides frequencies: - to deal with correspondence analysis - application in image studies
Comparative Scale
- respondents compare objects with respect to one or more stimuli - non metric character (nominal or ordinal) - it gets classification and preference relation types: -pairwise comparison scale - order (or hierarchical) scale - constant sum scale - continuous comparative scale - association scale - guttman scale - classification Q scale
Pairwise Comparison scale
- respondents compare two objects and chose one - to get frequencies (nominal scale) - transferable to an ordinal scale (preferences) - transferable to an interval scale applying "Thurstone's Law of Comparative Judgement" - ex: respondents compare (in pairs) two brands and chose one - 6 combinations of brands A, B, C, D Disadvantages: - the number of objects to compare is not high because the number of comparisons (n(n-1)/2) would be excessive. (6 objects > 15 comparisons) - no transitivity problems (ex: A preferred to B; B preferred to C; but C preferred to A)
Continuous Scale
- respondents mark the position between two extremes, reflecting their opinion with respect to an issue - they use a continuous (horizontal or vertical) line without other help or reference. disagree ----------------- agree - the researcher divides the line into categories and assigns scores to them -easy processing, but the score is confusing and unreliable
Classification Q scale
- respondents order a large number (60-140) of stimuli (brands, concepts) in a limited number of groups (ex. favorite to least favorite) according to criteria, - the number of stimuli in each group is specified a priori to obtain an arbitrary Q distribution - normal distribution (with symmetry and maximum number of stimuli at intermediate values) - each stimulus receives a qualifying order (1-5, 1-11, 0-10) --> original data Ex: Each individual in a group of 100, has to choose 9 newspapers from the 75 shoes. From the 66 remaining select the 9 which you like least. From the 57 remaining, choose the 15 which you like the most. From the remaining, choose the 15 you like the least. After this process, write the last 27 newspapers in the "I do not mind" column.
wording of questions should be inviting and enhance collaboration
- similar situation to a conversation between interviewer and interviewee - it is good to use a friendly tone, a neutral question and familiar words
Pairwise (transformation of the nominal scale into interval scale)
- the matrix of percentages becomes a matrix of standardized scores z~N(0,1) (the standard deviation values associated with a % of area under the Normal Curve) - the mean of z columns is an interval scale
Non- comparative scale (metric)
- the respondent evaluates each object independently, without providing references for comparison - metric scores (scale interval or ratio) Types: - continuous scale - item subjective scale
Considering the freedom of choice answers: Closed Question
- these proposed answers or fixed items. Useful in phone questionnaires with quick answers, and for people with low culture and insufficient memory. - They collect superficial information about facts or attitudes. Advantages: - Immediate coding and simple statistical analysis - the avoid doubts and misunderstandings. Disadvantages: - they can force the respondents answers - the do not detect unexpected answers
Step 1: Research Purpose
- to establish the problem or opportunity that the firm has to face or the alternative decision to take - in depth analysis (pilot study), collecting documentation to define the problem and objectives - WHY?
Steps in the development of a questionnaire
1. Research Purpose 2. Information needs 3. type of questionnaire 4. definition of information blocks 5. contents and wording of questions 6. analysis of difficulties answering 7. examination of the questionnaire strutcture 8. format and presentation 9. proof
Types of questions
1. considering the freedom of choice answers 2. considering how the question is constructed 3. considering the purpose of the question 4. considering the subject
Steps in a likert scale construction
1. to generate many relevant clauses for the attitude to be measured (qualitative research), classifying them a priori as positive or negative 2. to select a sample (100) of individuals (judges) 3. Rating of statement by the judges (pretest) and obtaining the total score by adding the values of each statement. 4. selection of the clauses most explanatory about the attitude (excluding vague or little discriminating ones)
Considering the freedom of choice of answers: Semi - closed answers
Closed questions with some open items that can add opinions not covered. - ex: In which financial institution is the account your salary is transferred to? 1. caja madrid 2. cam 3. bbva 4. santander 5. others (Specify)
Types of Variables
Continuous variables: These take an uncountable infinite number of values - Ex: weight, heigh -> in each pair of values there are intermediate values using a more accurate measuring instrument Discrete Variables: These take a finite or infinite countable number of values - Ex: number of children, family income Unclear distinction in practice: many continuous variables are treated as discrete (ex. age in years ignoring smaller units of time) This is because: - Measuring instruments have limited accuracy when observing continuous variables. - There are discrete variables that can take a number of high values (ex. family income)
Considering how the question is constructed
Indirect Questions: - They apply the philosophy of projective techniques when it is difficult to get an answer. - Ex: what do you think about people with religious beliefs? Direct Questions: - they get a direct answer - Ex: do you have a dog?
considering the purpose of the questions: classification questions
It classifies and then segments the respondents according to cultural, socioeconomic and personal criteria. It detects differences of opinion between the profiles and segments. Ex: Martial Status single married divorced widowed
systematic error
It constantly effects the measurement (the error always appears) - derived from the individual stable features, such as mechanical factors - ex. measure speed with an inaccurate speedometer
number of interviews
It depends on the size and heterogeneity of the population - 15 in short questionnaires - 30 in long and complex questionnaires
Step 3: Choosing the type of questionnaire
It depends on: 1. method of administering the questionnaire (face to face, phone, or mail) 2. structural level of the questionnaire 3. characteristics of the survey respondent (person or entity)
Considering the purpose of the questions: card questions
It requires a card to be answered. Applications: - Q. with different answer options - Q. with numerous answer options or difficult retention The prevent that the last answers are most likely to be remembered and chosen. Ex: Of those listed, which ones do you think are the main reason for the rise in prices. 1. the excessive number of tourists 2. inadequate government measures 3. excessive increases in salaries 4. etc 5. etc
multichotomous questions
Questions that give more than two options. -Single response: Which of the following brands of beer do you prefer? 1. budlight 2. budweiser 3. etc - Multiple choice: In times of economic crisis, what solutions are usually employed at your establishment? 1. to reduce time under 8 hours. 2. to reduce temporary staff 3. to reduce fixed seasonal staff. 4. etc
Considering the purpose of the questions: Subjective question scale
Respondents place themselves subjectively in the variables Scales: Very good..... very bad it like it very much ..... i dont like it well off....... modest ex: on a scale of 1 (not satisfied) to 7 (completely satisfied) what is your satisfaction with the public transport in this city.
Considering the purpose of the question: Box of battery of questions
Series of linked questions that complement one another in order to delve into an issue. They are presented in a two way table. These questions give fluency to the questionnaire. Ex: Which channel did you watch yesterday and what type of programs?
Considering the purpose of questions: Filler Questions
Simple and neutral questions, whose information is not required for the study Applications: - to start the interview - the change the subject - to improve the atmosphere in sensitive or controversial issues
Basic measurement scales
To measure: - to assign a real number or symbol to each object (products, companies, individuals) from a group according to certain rules, which pursue to obtain correspondences between features to measure (eg, flavors, income, satisfaction, usage) and to measure and the assigned numbers Measurement scale: - reference system which is established according to certain laws and allows measurements and comparisons of the phenomena.
Technique of questions with random answers
Two questions are asked simultaneously: irrelevant: - does your ID finish in a 4? - did you go to the cinema last week? delicate: - have you ever taken drugs? - have you taken someone's belongings or stolen products from a commercial establishment this week?
attribute
characterisic of a population not susceptible to being measured numerically (sex, marks) - it may have several forms, being the most simple attribute the one with two modes: presence / absence.
codes:
correct: from the following list of annual income, please tell me the number next to the section you belong 1 [less the 60,000] 2 [between 60,000 and 90,000] 3 [between 90,000 and 120,000] 4 [between 120,000 and 150,000] 5 [more than 150,000]
Variable
feature to observe from the elements of a population susceptible to taking numerical values (ex weight and height)
Questionnaire
form containing the survey questions and where the responses are recorded
format and presentation
getting a questionnaire that is attractive and easy to handle (especially in self administered questionnaires) and encourage answers.
Hierarchical relationship between basic measurement scales
hierarchical: - ratio (best properties) - interval - ordinal - nominal (least operational one) Moving from a higher level to a lower one. Not moving from a lower to a higher scale, except with approximations.
Concept measurement
measurement = true value + systematic error + random error
random error
non constant error - related to transitory circumstance or measurement situation - ex. interest of a person in answering a question - it is manifested in the lack of consistency of repeated measurements of the same person or object - it affects the reliability of the measurement.
types of basic scales of measurement
non-metric scales: to measure attributes (qualitative or categorical variables) - nominal - ordinal metric scales: to measure variables (quantitative variables) - interval - ratio
reliability vs validity
reliability is the congruence (concordance) level of the results of a test - reliability does not ensure the validity of a measure: - a measure can be reliable but not measuring what it intends to measure - reliability is a necessary, but not a sufficient condition for the validity
internal consistency
reliability test - it measures the degree of equivalence with a measurement
Test - retest
reliability test - it measures the stability of a measurement in time, if there are not any temporary changes in the characteristics - pearsons correlation test (coefficient) between similar measurements should be high
Questionnaire Design
requires combining: - common sense - experience - theoretical knowledge (steps in the development of the questionnaire)
venue (location)
similar to the environment that has to be studied
Precoding of a questionnaire
telephone and personal questionnaires have printed identification codes
item subjective scale
the respondent chooses one of the answers (ordered, limited, and expressed with numbers or descriptions) unpleasant - very - enough - little - nothing -little -enought -very Pleasent
questions about sensitive issues require being more careful when writing and asking them
they offend the sensibilities of the respondent and are more difficult to formulate sensitive issues: - tax and revenue issues - political and religious ideology and awareness - health and personal care - intimate or sexual conduct - breach of rules
Pretest or Proof (pilot survey)
things to consider: - objective - participants - venue (location) - number of interviews -pretest procedures - analysis of pretest
Objective
to analyze attributes of the questionnaire - questions (meaning-writing, difficulty, time to answer, interest and attention of the respondent) - sequence structure (order, filters, transitions between questions / sections) - instructions - presentation
To use clear and simple language
to exclude difficult words or concepts if the questions is addressed to people with different socioeconomic status or educational level (language must be appropriate for the level of the survey respondent)
Criteria for building scales
way to propose an answer - categories are indicated by: words, numbers, graphic elements definition and expression of the extremes of the scale obliged nature of the answer - neutral option (it requires an off number of answer alternatives DOESNT FORCE THE ANSWER) - non neutral option (includes DK?NA) forces the response Symmetrical or balanced scale - bidirectional with a neutral point and equal number of favorable and unfavorable options
difficult issues to answer
when individuals don't know exactly how they will behave regarding the issue WRONG: will you buy the product XXX in the coming months? yes/no CORRECT: Can you say to the degree of probability of the product X purchase in the coming months? Very unlikely - - - - - - - - very likely
to exclude words with emotional connotations
words with emotional connotations apart from their meaning - ex. red, communist, fascist, radical, capitalist, bourgeois, oligarch, people who do not work, WRONG: are you in favor of the death penalty for violent crimes?
battery of questions:
wrong: how many times do you brush your teeth per day? correct: - how long does a tube of toothpaste last at your home? - how many people use it? - what is its normal size?
Indirect questions:
wrong: what are your religious beliefs correct: what do you think of people with religious beliefs?