PR Research Quiz 3
formal scientific research
- empirical in nature or concerned with the world that can be experienced and measured in a precise manner - formal research methods produce results that are objective and values-free. This means the research results do not unduly reflect the biases of researchers or the attitudes and opinions of a few selected individuals but instead reflect events, facts, and behaviors as they exist or naturally occur in a population or group - typically these are members of a targeted audience in PR - Social scientists generally consider a research project formal to the extent it incorporates the characteristics of objectivity, systematic collection of data, representative samples, and replicable results into its design
analytical surveys
- explain why certain circumstances, attitudes, and behaviors exist among members of a specific population - this type of survey research usually involves advanced forms of statistical analysis to test hypotheses concerning relationships among a group of variables under study
personal interview variations: group-administered surveys
- group-administered surveys combine some of the features of personal interviews and some of the features of mail or Web-based surveys. Researchers give the survey to members of a group who complete the survey individually, usually with minimal input from a survey administrator - The research setting raises some concerns about group-administered surveys. Group members may feel coerced to participate, and if they fear their anonymity will be comprised, may not answer questions honestly.
data collection
- in many respects it is the beginning of the final phase of a research project - By this point, the researcher often has made the most difficult project-related decisions. Practitioners' level of involvement in data collection range from managing all aspects of data collection to leaving all aspect of data collection to a project manager or field-service provider.
results high in external validity or project ability
- in reality, no sample is perfectly representative - however, samples collected using probability methods are more trustworthy than other samples. This type of trustworthiness produces results high in external validity or project ability.
mixed-mode surveys
- many researchers now design surveys with more than one mode - the reasons it took so long for researchers to embrace mixed-mode research is that first it wasn't necessary in most situations and research indicates different modes of research can produce different answers to questions
rankings
- rankings change depending on the types of institutions included in the analysis, how the organizations collect data about "quality" or "value" and when they collect the data - rankings can change depending on whether they focus exclusively on schools that meet criteria for "national" universities or mix public and private colleges along with public and private universities
formal research methods
- require researchers to follow a systematic set of procedures providing for the uniform collection of data - scientific research results rely on representative samples. When representative samples are not available, investigators must understand the limitations of samples to the greatest extent possible. - when researchers use probability-based sampling methods and draw a sample of appropriate size, they help ensure that the behaviors and attitudes of sample members reliably represent the range of behaviors and attitudes found in a population.
survey sampling
- sample selection procedures depend on survey objectives. - sampling procedures range from convenient to complex. They also vary in terms of trustworthiness.
measuring a simple idea
- sufficient in number to represent enough about the concept - appropriate as indicators of the concept - unambiguous, so that a high score on a measure clearly represents a high level of the concept
descriptive surveys
- surveys generally fall into one of two broad categories: descriptive and analytical - descriptive surveys are used to document current circumstances and conditions and to generally describe what exists in a population
editing and coding
- the processes that research-team members use to translate the information collected in questionnaires into a form suitable for statistical analysis. - coding may be necessary when a questionnaire has open-ended responses or other data needing categorization. - coding may also be necessary to help prepare data for analysis.
survey research
- vital to organizations in a variety of different fields including all levels of government, political organizations, mass media corporations, educational institutions, entertainment conglomerates, and other product manufacturers and service products. - campaign managers may use research at all stages of the program planning, implementation, evaluation process. PR practitioners most commonly use survey research in the planning and evaluation phases of a campaign - an indispensable part of organizations' attempts to monitor their internal and external environments, solve complex problems, and plan and evaluate communication campaigns.
design features that affect response rate
- what the study is about - who the sponsor is - how the results will be used - why respondents should be a part of the study (how they were chosen and why they should care about this) - the extent to which their responses will be confidential or anonymous
probability sampling
- when used correctly and in combination with a properly applied survey method, usually provides an accurate and reliable understanding of the characteristics of a population. - the use of proper sampling methods is one of the most critical aspects of any research project and an especially important characteristic of scientific survey research
personal interview variations: computer-assisted interviewing
- with the advent of mobile technology, some research organizations conduct personal interviews with the help of laptop computers or mobile devices. - CAPI: occurs when research-team members conduct the interview. - computer assisted self-interviewing (CASI): occurs when respondents enter their own answers to questions - that is, complete a self-administered interview using a computer or mobile device - the main advantage of this type of interview is that computers aid in the standardization of survey administration and data collection - a primary drawback of computer assisted interviewing is its expense.
Don't assume
- Avoid asking questions that assume the respondents are familiar with the specifics of it. Ex: What do you think of the latest feature added to The Indiana Student's Website? What do your friends think about the latest curriculum proposal?
Avoid absolutes
- Avoid including absolutes in your item phrasing. - It's confusing to participants and responses won't be valid. Ex: I always read the daily news. (Agree-Disagree scale) Everyone thinks it is a good idea to double major. (AgreeDisagree scale)
Multiple-choice and Checklist
- Categorical (not scaled) response options. - Recommend having an "other (please list: ________)" option when you aren't 100% certain of all the response options to offer. - You'll then review and categorize the "other" responses during analysis.
4. Create codes (continued)
- Codes should be specific (by topic of RQ) - A quick example: Research Question: What influences a mom's decision to vaccinate her daughter? - Codes we created: - Influence—doctor - Influence—TV ads - Influence—daughter - Influence—prior experience
Effects on Analysis
- Doesn't make sense to report an average of categorical or ordinal measures - Can transform ratio/interval data into lower category, but can't take data from lower levels of measurement and transform it to higher level - Able to do more sensitive calculations on interval or ratio level data
5. Code and Contextualize
- Label quotes by your codes. - Contextualize your coding with notes. - Contextualize - To think about or provide information about the situation in which something happens - To consider information together with everything relating to it in order to understand it better
How to measure variable concepts
- Let's say you are interested in how much violence is shown in children's cartoons? - How would you measure violence? - Violence is the concept - # of times characters are shown punching, hitting, shooting... is the measure of violence shown - Attitude about Uber? - How would you measure attitude? - Attitude about Uber is the concept - Liking, disliking, ratings of good-bad, useless-useful, in comparison to other services (ranking), etc. are the measures of attitude
Review coding and context notes to tally, summarize, and interpret
- Look over your coding, notes, and quotes to answer each research question. - Discuss the answer to each research question by integrating your coding results. Which topics were discussed? How often? Any co-occurring? Were opinions on topics generally shared by all participants or what were the disagreements? Were any topics/opinions more/less intense or specific?
Checklist and Rank Order
- Not good if you intend to use the responses as part of a relationship comparison (analysis issues) - How does attitude about X relate to use of Y - Good if all you want is descriptive information
Common types of items
- Open-ended - Closed-ended - Multiple choice (choose one) - Checklist (check all that apply) - Rank order - Likert scale: "Likely Neutral Unlikely" - Semantic differential: "Happy............... Sad"
Ratio Level of Measurement
- Ordered measurement points with equal distance between them - True, meaningful zero
Interval Level of Measurement
- Ordered measurement with equal distance between options - No true zero point
Closed-ended Items
- Participants given a question or statement and a set of responses to select from - Pro: Creates easily comparable responses - Con: Response options must be known in advance
Ordinal Level of Measurement
- Ranks categories in logical numerical-based order - Sequence suggests value judgment - The order matters, but the difference between each element within the order doesn't matter, can't be measured, and/or is different
Order Effects Example
- Rate the importance of the following free services offered by Health Center (then list...) - How much do you agree with the following statement, "Health Center offers a variety of services." - Are you aware that Student Health offers STI testing services?
Surveys. A Quantitative Method: Weaknesses
- Sometimes lack context - Inflexible - Artificial - Lack ability to follow-up - Response set choices can force round pegs into square holes
Writing Your Own Questionnaire continued
- Straightforward and clear - Respondent should know how to answer - Avoid abbreviations and slang expressions - Shorter is better than longer (only ask what's relevant) - Select appropriate item format and response options - Group like items together - Provide sufficient instructions for each type of item Important to pretest your survey before you start collecting data
Likert Examples
- Strongly agree - Agree - Neither agree or disagree - Disagree - Strongly disagree
Qualitative analysis should be...
- Systematic = deliberate, orderly and structured - Not impulsive or arbitrary - Can still be flexible - Well-documented = write down notes about decisions made, interpretations, and how topics are defined - The researcher should be able to point to a trail of evidence (verifiable) and logic supporting all these issues
frequency scales
- The frequency scale is an interval or ratio scale. Instead of assuming how much a respondent embraces an idea or opinion, the frequency question ascertains how often the respondent does or thinks something - Sometimes frequency scales are constructed in ways that make it unclear whether equal distances exist between each response category, which makes the meaning of the measure less clear and the assumption of interval-level statistical power questionable.
Measurement
- The types of items and response options you use to measure your data determines what types of questions you can answer with your data - (& the types of analysis you can do)
Question Formats: Open-Ended Questions
- There are no response limitations - Respondents provide their own answer options - Good for exploring new and different information
Question Formats: Closed-Ended Questions
- There is a limited or fixed number of response options for respondents to chose from - Are mutually exclusive - Dichotomous questions have two options
Open-ended Items
- Try to minimize the use of these - Code responses after all data are collected
Surveys. A Quantitative Method: Strengths
- Useful in describing large populations - Standardized!Generalizability (*sampling) - Modalities - Range of analysis techniques are possible
Important issues to consider when writing questionnaires
- Validity and reliability concerns as they relate to the sample, the topic and the client - Levels of measurement and why they matter - Ways to ensure clarity and avoid bias - Types of questions and how the information each type provides differs - Questionnaire layout and design to ensure logical flow and visual clarity
Valid and Reliable Measures
- Validity: Are you measuring what you think you are measuring? - Reliability: Can you measure it consistently?
Order Effects
- Warm up participants - Sensitive items should be last - Always consider: Will items that you are asking influence responses to following items?
Don't make it easy to have no opinion
- When you give respondents the opportunity to answer a question with "no opinion," "I don't know," or "not applicable," many will pick that easy option. - Do the hard work up front and ensure you've provided them with sufficient responses to choose from.
Don't use leading or loaded items
- You don't want to lead your respondents into answering a certain way based on the wording of the items. Ex: Did you do your patriotic duty and vote last Tuesday? How much do you agree that education is important?
personal interview variations: mall-intercept surveys
- a common method of collecting data in a personal interview format. - different locations for intercept studies are malls, downtown areas, college campuses, and other areas that attract large numbers of people - the most important benefit is researchers can complete a study relatively quickly if necessary, and mall intercepts are inexpensive relative to other research methods - the most important limitation is to generate a probability-based sample in an intercept study is difficult as a practical matter
experiments
- an experiment involves taking an action and observing the consequences of that action. - In a sense, experiments are a natural part of PR because in most cases, practitioners take action of one sort or another (researchers call this a treatment) and then gauge the effect of that action on a targeted audience members' attitudes or behavior.
To avoid obtaining misleading results
...questions must be clear, must elicit honest and reliable answers, and must keep the respondent interested in providing answers The construction of questions must differ according to whether the questions are being read or heard, and the researchers can ask only as many questions as respondents have the time and energy to answer. Guidelines for questionnaire design typically focus on the importance of clarity, simplicity and objectivity. Other important considerations include making questions interesting and letting the questionnaire progress logically so that respondents feel motivated to answer carefully.
Closed-Ended Questions
1) Have you completed a survey before? Yes or No 2) What is your gender? 1) Male 2) Female
Contingency Item
A survey item intended for only some respondents, determined by their responses to an earlier item.
What is a Survey?
A system for collecting comparable information across many people
Don't ask impossible to answer items
Avoid asking items that are beyond participants' capability to answer. Ex: How many hours of TV do you watch in a typical month? What did you have for breakfast Jan. 17th?
characteristics of telephone surveys
Benefits - Relatively inexpensive (reasonable cost per completed interview) - Data collection can be completed quickly - Reaches widely dispersed sample members relatively easy - Lists or random digit dialing make it possible to generate probability-based sample - Rapport established with respondent can help gain compliance Limitations - interviewer bias may occur - not every household has a land line telephone and cell-only users are less likely to participate (potential source of bias) - households with land lines are likely to be equipped with answering machines and/or caller identification - short, relatively simple questionnaire required - limited interview flexibility - respondents may not answer thoughtfully
characteristics of online surveys
Benefits - generally inexpensive - many population subgroups have near 100% Internet access including professionals, college students and government employees - reaches widely dispersed sample members easily - nearly immediate delivery and potentially quick data collection - no interviewer bias Limitations - Internet access still is limited; it is impossible to conduct a probability-based survey of the U.S. general population, for example - Potential participants need reasonably equipped computer and basic computer competence - It is difficult for researchers to encourage survey participation - Volunteer samples are likely to produce results low in generalizability - May suffer from very low response rates
characteristics of mail surveys
Benefits of mail surveys: - relatively inexpensive - reaches widely dispersed sample members easily - mailing lists make it easy to generate probability-based sample - may provide high degree of anonymity (potentially useful for sensitive topics) - no interviewer bias Limitations of mail surveys: - may suffer from low response rates (requires inducements and multiple mailings to improve) - data collection may take longer than other methods - no questionnaire flexibility; typically requires a shorter, self-explanatory questionnaire - survey respondent may not be selected sample member - members of certain groups less likely complete questionnaire
survey critique
Benefits of survey research: - can be relatively inexpensive (researchers can control costs using less-expensive methods) - can have relatively quick project-completion time - practitioners can implement surveys relatively easily - can provide information concerning a large number of variables - useful for describing diverse populations - surveys can produce highly accurate, reliable results Limitations of survey research: - do not allow researchers to determine causation - some survey methods are expensive - data collection may take a long time - sometimes survey results lack accuracy and reliability - participation rates have fallen
characteristics of personal interviews
Benefits: - Interviewers establish rapport with participants - Often results in high-quality data - Bias from sampling frame often is low - High degree of interview/questionnaire flexibility Limitations: - Often expensive; typically highest cost per respondent - Requires high degree of administration and interviewer training - Data collection may take a relatively long time - Strong potential for interviewer bias - Some sample members difficult to reach
5. Code and Contextualize (continued)
Contextualize your coding with the following notes: - Frequency - Intensity - Specificity - Co-occurrence - (Dis)agreement
Correlation coefficient
Correlations are useful for examining the strength and direction of relationships between two interval-level (ex. very interested, not at all interested) or ratio level (ex. 0 times to 7 times) variables. Correlations are less intuitive for a client without a statistical background however. A correlation coefficient ranges between -1 and +1. A - 1 indicates that the two variables are exact opposites.
Mode Comparison
Cost per response: Internet: Lowest Phone: Medium Mail: Low Face-to-Face: High Speed until sample collected: Internet: Fast, Phone: Fast, Mail: Slow, Face-to-Face: Medium Response rate*: Internet: Poor, Phone: Okay, Mail: Okay, Face-to-Face: Excellent Clarification possible: Internet: No, Phone: Yes, Mail: No, Face-to-Face: Yes Social desirability likelihood: Internet: Less, Phone: Medium, Mail: Less, Face-to-Face: Most Interviewer influence possible: Internet: None Phone: Medium Mail: None Face-to-Face: Most
levels of measurement
1. First level is nominal, referring to the names or categories of things. This level of measurement can show how many people fit into particular categories. - "Which of the following companies do you find credible?" 2. The second level is ordinal, indicating that some meaningful order exists among the attributes. These questions should have answers that are mutually exclusive, exhaustive and ordered in some way. (pg. 199) - "Rank the following companies from most credible to least credible, with the most credible company receiving a 5 and the least receiving a 1" 3. Third is interval level. This is the most flexible type of measure to use because it holds a lot of meaning, giving it a great deal of explanatory power and lending itself to sensitive statistical tests - "How credible are the following companies? From not at all credible to very credible 4. Fourth is ratio scale, which is simply an interval scale that has a true zero. This means the numbers assigned to responses are real numbers, not symbols representing an idea such as "very much."
measuring knowledge
1. First option is to give a true/false or multiple-choice test 2. Second option is to ask open-ended questions in which people must fill in the blanks 3. A third option is to ask people how much they feel they know, rather than testing them on what they actually know 4. Fourth option - Follow-up questions can ask people how sure they are of a particular answer
survey planning process
1. Identify goals and objectives for the project 2. Establish sample selection procedures 3. Determine survey method 4. Design questionnaire 5. Collect data 6. Edit, code and compile data 7. Analyze and interpret data
mail survey timeline
1. Pre-notification card: typically received 1 week before the questionnaire 2. Cover letter and Questionnaire: sent out following the pre-notification card 3. Thank you card or letter: sent out 1 week to 10 days after the cover letter and questionnaire 4. New letter and Questionnaire: typically sent 2-3 weeks after the arrival of the initial questionnaire 4. Additional questionnaires (optional): some researchers will send additional cover letters and questionnaires to give potential respondents more opportunities
Steps for Qualitative Data Analysis
1. Prepare transcripts for analysis 2. Make two copies of each transcript 3. Read all transcripts in one sitting 4. Create codes by research question 5. Code with RQ codes and contextualize 6. Review coding and tally, combine, summarize, interpret
two important concepts of measurement
1. Reliability 2. Validity
components of reliability
1. the indicator of a concept must be replicable, that is, reusable with a similar result 2. The second component of reliability is how consistently the various operationalizations of a concept measure it
Semantic Differential
A response option format in which the respondent is asked to rate something in terms of two opposite adjectives (or antonyms).
Disagreement or agreement
Disagreement or agreement may occur for a topic - Among participants or even if someone contradicts themselves
Don't use double negatives
Double negatives make respondents have to think harder, which makes them impatient, which means they're more likely to provide inaccurate answers (or quit). Ex: Which of these pictures is not unattractive? Do you disagree with those who do not want to build a new swimming pool?
Don't use double barreled items
Double-barreled items ask two questions at once. In a survey, multiple ideas presented at the same time inhibits a useful response. Ex: How interested would you be in learning STI statistics and reading a personal story about getting tested? How often do you feel stressed and sad?
Categorical Level of Measurement
Favorite music genre 1. Pop 2. Hip hop 3. Jazz 4. Country western 5. R&B 6. Folk 7. Electronic
4. Create codes by research question
For each research question, create a main code and your anticipated sub-codes (based on your earlier reading of the transcripts). Also, create an "other" code to capture interesting/important information that isn't specific to an RQ, but still may be relevant.
Longitudinal
Gathered over multiple time points (best to show trends, changes)
ease of reading in questionnaires
It helps give respondents chunks of questions at a time. A series of questions without a break can become boring and confusing. People may get lost in a written questionnaire that has 10 items in a row
What is responsible for the rapid increase in the use of survey research?
Organizations have increasingly felt the need to understand the opinions, attitudes and behavioral motivations of their key target audience members, including legislators and government regulators, community members, consumers, employees, and other important groups
close-ended questions: ranking scales
Ranking scales are ordinal variables, in which respondents are asked to put a set of items in the order they think is most appropriate. They are problematic because they require respondents to perform a complex and often confusing task.
1. Prepare transcripts for analysis
Recommend: Single-spaced comments and then double space between speakers. Label by source (based on what's relevant to you). The comments of the moderator should be easily identifiable by bolding, caps, or underlining.
Chi-square
Statistics such as the chi-square can be used to highlight especially notable differences across groups. Many clients however do not want to see the statistics themselves and may find too much statistical info intimidating.
How to write items
Step 1: See how existing surveys ask about what you want to know
Writing up your qualitative findings
Summarize your findings by research question. Label these sections. - Note: Avoid percentages (70% of interview participants said) instead write in more qualitative terms (most interview participants discussed). - As much as possible, insert a quote from your data that supports your finding summaries.
edge coding
The behind-the-scenes use of codes to provide important contextual information is known as edge coding. These codes include: 1. Tracking information 2. Question information 3. Answer codes
determining the data collection method
The primary means of collecting survey data include personal interviews, mail surveys, telephone surveys, and electronic surveys conducted via the Internet. In addition, researchers may combine methods to conduct a mixed-mode survey.
To overcome, provide "face-saving" options.
We know that sometimes things prevent individuals from voting that may be out of their control. Which of the following statements best describes you? - I did not vote - I thought about voting, but didn't this time - I usually vote, but didn't this time - I am sure I voted
clarity of graphics
Work by Christian and Dillman has shown that respondents to self-administered surveys pick up important cues from the visual design of survey questions and answers. For example, Christian and Dillman demonstrated that respondents become confused if a scale is broken up into two columns instead of being presented in a single row or in a single column
constructs
abstract ideas that are not observable
encouragement
because respondents to telephone surveys cannot see the questionnaire, they will be worrying about how long the interruption will take. If the questionnaire seems to go on too long, they will lose interest. As a result, it helps to thank them every so often for continuing
pictoral scales
can be useful for special populations such as children, individuals lacking literacy or populations with whom language is a difficulty Often the scales range from a big smiley face (very happy or positive) to a big frowny face (very unhappy or negative), or from a big box (a lot) to a little box (a little)
Levels of measurement: Continuous
can take on any value: - Ordinal* - Interval - Ratio Measurement level affects how you can analyze the data.
best research method
despite what some have suggested, there is no best method of survey research, and the potential advantages and disadvantages of each method do not apply equally, or even at all, to every research situation. The best method for a project depends on various situation-specific factors, and because of this, project managers must consider the use of a survey research method in relation to the needs and constraints of each situation.
Levels of measurement: Discrete
finite set of values - Categorical/ nominal - Measurement level affects how you can analyze the data.
Two types of personal interviews: structured
in a structured interview, interviewers ask questions in a predetermined order and have little freedom to deviate from the questionnaire, also called a survey schedule. the result is an interview process that often produces high-quality data, has a reasonable response rate, and lends itself well to various topics and question types.
Two types of personal interviews: unstructured
in an unstructured, in-depth interview, interviewers ask broad questions and give respondents freedom to respond as they wish. The results of these interviews are challenging to analyze because of the unstructured nature of the interview.
pre-campaign surveys
in the campaign planning phase, pre-campaign surveys help practitioners establish benchmarks (the pre-existing standard against which they will evaluate the success of a campaign) so they can set campaign goals.
telephone survey introduction
introductions by telephone need to be even shorter than those by letter. If possible, keep the introduction to two sentences.
Frequency
keep a tally of how often a topic is mentioned
cultural/language sensitivity
knowing the target public well can aid design and secure a better response.
Intensity
look for words that connote intensity of feeling about a topic (strongly held opinion or not)
mail survey introduction
must have a cover letter introducing the study. It should be brief (never longer than one page), on a letterhead (to identify the sponsoring institution and lend credibility to the study), include a contact person for questions or concerns, indicate how long the survey will take to fill out (be honest) and make it clear how to return the questionnaire.
length
no perfect length exists for a questionnaire, although a good bit of research focuses on the topic. For example, people seem likely to participate in a mall-intercept survey limited to a 5" x 8" card. It is difficult to keep people on the phone longer than 5 to 8 minutes
nonresponse error
occurs when members of a sample fail to complete a questionnaire. By not completing a questionnaire, non-respondents potentially are introducing error or bias into survey results.
coverage error
occurs when not every member of a population has an equal (nonzero) chance of being included in a sample. A survey's results cannot represent everyone in a population if some people are systematically excluded from the sample. Increasing contact and participation using a mixed-mode study is likely to help reduce coverage and nonresponse errors.
a valid measure
one that seems to represent a particular idea in a convincing way For measures to be valid, the concept or idea they represent must be clear and the operationalizations, the actual measures themselves, must seem appropriate
opt-in internet surveys and panel surveys
panel studies are a type of longitudinal survey that permit researchers to collect data from the same participants on a repeated basis. Because of this, panel studies allow researchers to examine changes in sample members over time. This differs from standard surveys, which are cross-sectional in nature. As cross-sectional research, a survey provides an immediate picture of participants' opinions and attitudes as they currently exist but little information about how participants formed these attitudes or how they change over time. A strength of panel studies is their ability to provide researchers with info concerning how participants' attitudes and behaviors change as they mature or in response to specific situations.
between-campaign research
post-campaign research also can serve as between-campaign research. That is, many organizations simply transition from one or more existing campaigns to new campaigns without stopping to conduct new research at every point between campaigns.
post-campaign research
practitioners use post-campaign research as part of the campaign evaluation process to help them determine whether a campaign has met its goals and related purposes.
questionnaire design
proper questionnaire design contributes significantly to the trustworthiness of survey results. Good survey questions, when combined with appropriate data-collection methods, produce accurate responses. Poor survey questions or inappropriate data-collection methods produce untrustworthy results that can misinform PR managers poorly designed questionnaires often bias participants' responses
open-ended questions
questions on surveys are usually close ended, meaning respondents choose their preferred answer from a list of possibilities. Open-ended questions, which ask a query but provide space for individual answers instead of a response list, invite more information but are often skipped by respondents and are time consuming to analyze afterward. As a result, surveys typically limit the number of open-ended questions 2 to 3 out of 50.
directionality
refers to the match between the numbers that represent answers to questions in the computer and the idea they symbolize.
researchers must consider...
researchers must consider the population and sample, including the best way to contact sample members, the sampling methods available, the survey topic or topics, and the importance of reliability and validity
replication
researchers should be able to reproduce the results of formal research projects.
Specificity
responses that are specific and based on experiences should be given more weight than vague/impersonal responses
sensitive questions
sensitive questions should never appear at the beginning of a survey. Instead, the most sensitive questions come at the end so respondents won't quit the survey if they're offended; this is why demographics go at the end.
using "skip patterns" effectively
sometimes a question will not apply to all respondents. In this case, researchers use a screening question, or a series of screening questions. This series of questions is known as a skip pattern.
intermediate campaign evaluations
sometimes organizations conduct surveys and other forms of research during a campaign to provide intermediate campaign evaluations. This monitoring helps campaign managers determine whether a campaign is on course. In such a case, they use research results to monitor campaign progress and to make connections in campaign strategies and tactics.
analysis and interpretation
statistical analysis and interpretation are the 7th step in survey research.
reliability analysis
tests a measurement tool to see how reliable it is
validity
the extent to which a measurement tool measures what it is supposed to measure how well the tool measures what it is supposed to measure
face validity
the extent to which a tool appears to measure what it is supposed to measure
construct validity
the extent to which a tool measures an underlying construct
content validity
the extent to which an individual items on a test are relevant to the content area being measured
predictive validity
the extent to which responses on a measure can predict certain behaviors
Univariate relationship
the minimum information usually required for a research report includes frequencies, percentages, means, and, for some clients, standard deviations. The frequencies tell the client how many people answered each question using each response. Frequency tables usually include percentages as well, so the reader can make informed comparisons across questions Researchers can also present frequencies visually, using tables, bar charts, histograms, or pie charts. Bar charts often can give clients a better grasp of the range and strength of responses than they might get from just a recitation of mean and standard deviation Pie charts, meanwhile, can communicate the contrasts and similarities between the usefulness of information sources.
survey planning
the most important aspect of survey planning involves identification of the purpose of a research project. This normally involves identifying a research problem and the potential hypotheses and/or research questions a project will address.
psychological measurement
the process of assessing/measuring psychological traits
data cleaning
the process of fixing mistakes in the data set
pre-notification cards or letters
the use of pre-notification cards to tell respondents that a survey will be happening soon helps boosts response rates.
operationalizations
the ways the researcher measures an idea, such as by counting the number of adults employed in the community during a single year.
call record
this is a record of the result of each call attempt, and it provides project managers with information they need to determine sample members who receive additional call attempts and to determine the response rate for a survey.
mail surveys
traditional mail surveys are conducted by sending a questionnaire via regular mail to a sample of individuals. A mail survey is a self-administered questionnaire.
close-ended questions: quantity/intensity scales
typically is an ordinal-or-interval level variable in which respondents choose a location on the scale that best fits their opinion on a list of options that forms a continuum.
interviewer directions
untrained or confused interviewers can ruin a survey. Interviewers must sound enthusiastic, polite and confident, presenting questions clearly.
incentives
usually the incentive is provided ahead of time to motivate the person to respond, instead of afterwards as a reward. Monetary incentives ranging from $2 to $5 are especially popular, with amounts over $10 rare. Other incentives can include a gift certificate or product samples.
reliability
when a measurement tool consistently gives the same answer the consistency of the results of a measurement tool how consistent the outcome of the tool is 1. Traits of the subject 2. Testing conditions 3. Chance Factors ^ all can cause inconsistent results
response set
when constructing a questionnaire, it is important to scatter questions that measure a concept instead of clustering them together. The reason for this is that people's answers can suffer from response set, which means they answer a set of questions similarly because they are answering too quickly or not thoughtfully enough rather than because they think similarly about each question in the set.
personal interviews
when investigators conduct personal interviews, they may invite respondents to a research firm or collect information in a respondent's home or office.
Response Rate
% of surveys that are completed (divided by how many people had the opportunity to take it) - Asked (sent to) 4,000 and 345 completed it - 345/4000 = 8.6%
Improving reliability
1. Increase the sample size 2. Control the testing conditions 3. Run a reliability analysis
response rates and outcomes - pg. 229-232
1. The total sample size 2. The valid sample size 3. The completion rate 4. The response rate 5. The refusal rate 6. The noncontact rate standards for minimal disclosure: page 232
training interviewers
1. Use the questionnaire carefully, but not informally 2. Know the specific purpose of each question 3. Ask the questions exactly as they are written 4. Follow the order indicated in the questionnaire 5. Ask every question 6. Do not suggest answers 7. Provide transitions when needed 8. Do not leave any question blank
When designing questions, keep the following principles in mind
1. Use words that are simple, familiar to all respondents, and relevant to the context 2. Aim for precision to make sure the meaning of answers will be clear 3. Check for double-barreled questions 4. Check for leading or loaded questions 5. Check for social-desirability effects 6. Provide enough context to enable people to respond realistically or remember accurately In general, questionnaire designers must avoid yes/no items
3 possible relationships of reliability and validity
1. a measure can be reliable, but not valid 2. a measure can be neither reliable nor valid 3. a measure can be both reliable and valid
Cross-sectional
A one-time snapshot of a participant
online surveys including opt-in and panel surveys
A variety of options exist for contacting potential respondents when conducting electronic surveys and the best choice largely depends on sample members (pages 178-181)
How you ask the question, can change the measurement level
Categorical Level What is your favorite color? - Red - Blue - Green - Purple Ordinal Level Rank each color with 1 being your favorite and 4 being your least favorite. ___ Red ___ Blue ___ Green ___ Purple
co-occur
Certain topics may co-occur (e.g., whenever insurance is mentioned, cost is likely to be mentioned)
Goals of a Survey: Describe
Describe a "snapshot" of current attitudes/beliefs/behaviors - What percentage of IU students have used non-prescribed prescription drugs? - Do students view the risks of recreational Rx drug use as serious?
Goals of a Survey: Explore and Explain
Explore and Explain: Relationships between variables - How is X connected to Y? - Are freshmen less likely to use Rx drugs than seniors? - How do males react to fear appeals about recreational Rx drug use compared to females? - Is concern about GPA associated with intentions to use Adderall?
handling "don't know" responses
It is important to consider two issues here: the first is that "don't know" can be a meaningful response, of great usefulness to the communication manager. Another important issue about "don't know" responses is that "don't know" cannot be interpreted the same way as "neutral."
Order the questions
JUST LIKE A STORY HAS A BEGINNING, A MIDDLE AND AN END, SO DOES A SURVEY INSTRUMENT
Reporting Relationships among Variables
Often, a client wants to compare results for several groups, such as by education levels, sex, ethnicity, or membership. The crosstab table usually provides the most intuitive presentation format. Crosstab tables usually present frequencies and column or row percentages. The layout of the table will encourage readers to interpret the data in a particular way.
2. Make two copies of each transcript
One stays intact for reference One is used in analysis—upload on Canvas
Do: Help people to not feel judged
Social desirability bias: Over-reporting desirable behaviors, underreporting undesirable behaviors
How do you review coding?
Some people eyeball it. Some people cut and paste quotes into docs sorted by RQ.
close-ended questions: checklists
The checklist is a nominal variable, providing categories from which respondents can choose. They can be asked to choose only one response, or all that apply.
close-ended questions: likert-type scale
The most frequently used scale is known as the Likert scale (strongly agree, agree, disagree, strongly disagree)
3. Read all transcripts in one sitting
This quick reading is just to remind you of the whole scope and to refresh your memory of where information is located, what information is missing, and what information is common.
code book
a copy of the questionnaire with annotations that direct data entry personnel and analysts.
feeling thermometer
another variation of the Likert scale, which can be modified to measure levels of confidence, degrees of involvement and other characteristics
push polls
an insidious form of negative campaigning disguised as a political poll that is designed to change opinions, not measure them
semantic differential scales
an interval-level variable, on which respondents locate themselves on a scale that has labeled end points. It is useful to have at least four response options. can provide a lot of information in a concise format. Some consider these items more valid than Likert scale items.
confidential or anonymous
anonymous - no one, including the researchers, will know respondents' identities confidential - researchers will have identifying information for some purpose
Directionality and Response Set
another issue that can affect validity is known as directionality and refers to the order in which response categories are presented. It helps both respondents and analysts to associate negative opinions with lower number and positive opinions with larger numbers.
Questionnaire Outline
•Introduction script (be sure to indicate duration) •Main items •Demographic information
Writing Your Own Questionnaire
•Straightforward and clear •Respondent should know how to answer •Avoid abbreviations and slang expressions •Shorter is better than longer (only ask what's relevant) •Select appropriate item format and response options •Group like items together •Provide sufficient instructions for each type of item
