SPEA V 379
Collecting Survey Data Disadvantages
Are more impersonal for participants, and usually not good for getting quotes in participants' own words
Rating Scale: Norm Referenced Scales
Asks for a comparison with some reference group in mind - Above average, below average, etc.
What strategies are used to decrease the costs of responding to a survey?
Avoid subordinating language Emphasize similarity to other requests
Limitations of Focus Groups
Less control over the course of the discussion Data are more complex to analyze Moderating requires training Large variability between groups Requires more effort to assemble More tailored environment required
Predictive Validity
The extent to which performance on one measure predicts performance on another
Concurrent Validity
The extent to which there are correlations among different measures of a construct.
Example of Ratio
Through the end of 2016 spring semester how many course credits had you earned at IU?
social exchange theory
the theory that our social behavior is an exchange process, the aim of which is to maximize benefits and minimize costs
survey is created that will only take 5 minutes to complete. Social Exchange Theory says
this will decrease social costs
purpose of collecting data on observations
to gather information on how the program operates, or on visual changes that have resulted.
Purpose of collecting Survey/questionnaires
to get information from the individual about their changes in knowledge, attitudes and behavior
Collecting Survey Data Advantages
valuation Can get information from a lot of people Can be done confidentially or anonymously, so may be more valid Can use questions from existing surveys More effective when using yes/no or true/false type questions
Nominal (categorical)
Objects cannot be ordered or added; they are labels.
Ordinal (ranked)
Objects of a set can be rank-ordered; equivalence between objects not guaranteed.
Types of Reliability
1) Instrument: extent to which the instruments give repeated results 2) Ratings/ Observation: extent of which different raters would code an observation the same way 3) Instrument Items: extent which related items on an instrument access the same construct 4) Construct Scales: extent which results are consistent across two different scales
Types of Validity
1) Statistical Conclusion: Establishes strength of the co-variation of cause and effect variable *2) Internal Validity: the extent to which an experiment shows convincingly that changes in behavior are a function of the independent variable and not the result of unknown variables.* 3) Measurement: how well do the measures capture the observable characteristics of a real world phenomena *4) External: extent which conclusion of the evaluation can be generalized to other time and places*
respondent background/demographics
CHARACTERISTICS of the respondent
Common Mistakes when writing survey questions
Choose simple words over specialized words Use complete sentences to ask questions Avoid vague quantifiers when more precise estimates can be provided Avoid specificity that exceeds an accurate answer Use equal numbers of positive and negative categories for scalar questions Avoid bias from unequal comparisons Don't use "all the apply" formats Develop response options that are mutually exclusive Avoid double barreled questions Avoid respondents to say yes in order to say no
Rating Scale: Criterion-Based Scales
Comparison against the respondent's absolute criteria for performance - Excellent, good, fair, poor, etc.
Collecting Interview Data Disadvantages
Data can be difficult and complex to analyze Interviewer can bias the responses.
Pie Charts
Dont show comparisons between data series well
what types of question are appropriate to appear as the first question on a survey instrument?
Easy to answer questions Should be relevant to the stated purpose of the survey
Systematic Inquiry
Evaluators conduct systematic and data-based inquiries Adhere to highest technical standards Explore strengths and shortcomings of evaluation questions and approaches Communicate approaches, methods and limitations accurately
Integrity/Honesty
Evaluators display honesty and integrity and attempt to ensure them throughout the entire evaluation process: Negotiate honestly with clients and stakeholders Disclose values, interests and conflicts of interest Represent accurately methods, data and findings Disclose source of request and financial support for evaluation
Competence
Evaluators provide competent performance to stakeholders: Possess appropriate skills and experience Demonstrate cultural competence Practice within limits of competence Continually improve competencies
Respect for people
Evaluators respect security, dignity and self-worth of ALL stakeholders. Understand evaluation context Get informed consent and protect confidentiality Maximize benefits and minimize harm Foster social equity Respect differences among stakeholders
General and public welfare
Evaluators take into account general and public interests Include relevant stakeholders Balance client and stakeholder needs Examine assumptions and potential side effects Present results in understandable forms
Collecting Interview Data Advantages
Excellent for both process and outcome evaluation, because you can get in-depth information from participants Can ask sensitive questions that require confidentiality
Collecting Data: Focus Groups Advantages
Good for outcome evaluation, because you can ask people to explain how the program affected them Can identify a lot of issues and effects Can give staff better understanding of the program from participants' own words
Collecting Data: Observational Groups Advantages
Good for process evaluation, because you can get an immediate impression about how the program is progressing
Focus Groups
Group of people (usually 7-10) Participants share some common characteristics Produce qualitative data - related to perceptions, attitudes of consumers about products, services, or opportunities (not intended to develop consensus or make decisions) Involve a focused discussion (carefully predetermined and sequenced)
Collecting Data: Observational Groups Disadvantages
Information can be difficult to interpret May be biased by how staff interpret the information Events that are seen can be time-dependent, and not representative of what is really going on If people know they are being observed, they may act differently than usual
Advantages of Focus Groups
Place people in real-life social situations (where they are influenced by other's opinions) Allow moderator to probe (or dig deeper) for information. Have high face validity Have relatively low costs Provide speedy results Allow for the inclusion of more respondents (compared to interviews)
Purpose of the Guiding principles
Promote ethical evaluation practice Foster continuing professional development Stimulate discussion within and outside evaluation Evaluators aspire to provide best possible information that might bear on the value of whatever is being evaluated
Informal Interviews Definition
Questions may vary between respondents, very dependent on interviewers conversational skills. large amount of flexibility
Interval
Rank-ordered scale in which the intervals represent equal distances; no absolute zero value.
Ratio
Rank-ordered scale with equal intervals and an absolute zero value.
Collecting Data: Focus Groups Disadvantages
Requires a good facilitator Takes time to analyze and interpret the discussion May require extra resources for facilitator's time and participant incentives
Implication of Survey Research
Reward Cost Trust
What elements are part of Social Exchange Theory
Reward, Trust and Cost
To assess knowledge, attitudes, and behaviors, which response options are most appropriate?
Strongly agree, agree, disagree and strongly disagree
What strategies are used to increase the rewards of responding to a survey?
Show positive regard Say thank you Support group values
Content Validity
Similar to face validity, but relies on experts in the field to determine the level of representation.
What strategies are used to increase trust among survey respondents?
Sponsorship from a legitimate authority Emphasize the task's important
Social Exchange Theory
The actions of individuals are motivated by the return they are expected to bring from others: ex: taking surveys
Instrument Reliability
The extent to which a measurement instrument gives the same reading if repeated in a given situation. EXAMPLE: A researcher is trying to measure eating habits among female collegiate athletes. During the pilot phase of the study, participants are recruited to complete a 24-hour food diary during week 3 of the semester. They also complete a 24-hour food diary during week 5 of the semester.
Inter-Rater Reliability
The extent to which different raters would code an observation or rating the same way. EXAMPLE: In order to determine if a classroom observation scale yields similar results for all members of the evaluation team, results are compared across a number of different observers.
Face Validity
The extent to which evaluation stakeholders agree the measure represents the construct (people who are looking at this generally agree that this is a good measure).
Validity vs Reliability
Validity: - does it do what it's supposed to do? does it work? Reliability - the degree to which the tool produces consistent/stable results *Tools can be valid and not reliable (and vice versa)
Example of Ordinal (ranked)
What is your current year in school 1.Fresh 2. Soph 3. Junior 4. Senior
Under what circumstances would standardized interviews offer more advantages than unstructured interviews?
When looking for basic information When asking a large number of people you want the questions to be the same to summarize the data easily. How many people are conducting the interviews What extent do you expect the results of the survey to fall into different categories?
How can data collected through focus groups compliment data collected through more quantitative data collection strategies such as surveys?
by helping identify thinking patterns and learn related vocabulary
Use Bar Charts when
comparing values, displaying quantitative values that relate to each other
Informal Interviews
conversational (unstructured)
response error
design the survey instrument to include well-written questions that are well-laid out. Clear and concise questions
standardized interview definition
everyone receives the same interview from a list of arranged questions. can be useful when interviewing a large number of people
knowledge questions
factual INFORMATION the respondent is familiar with ex: "are the following statements true or false"
Sampling Error
has to do with appropriate sample sizes
Sample Bias
has to do with representative samples
purpose of collecting data on focus groups
hold discussions with groups of 10-12 people go understand in-depth what they believe were the effects of the program, or how they saw the program implemented
focus groups can follow quantitative measures by
interpreting data that comes from a questionnaire, or to develop action strategies for concepts identified by the survey
Semi-structured
interview guider
Standardized Interview
interview protocol
Semi-structured interview definition
involves a list of questions that will be explored during an interview. fair amount of flexibility
Purpose of collecting Interviews
obtain information from individuals about their experiences, or to learn more about their answers to survey questions
what types of data are pie charts most likely to be helpful
part to whole data that relates parts of something to the whole
what type of information is best gathered through focus groups?
qualitative data
feeling questions
respondents EMOTIONAL response related to particular experiences or thoughts
use line charts when
showing relationship between variables and time
Example of Nominal
what is your major? _____
behavior questions
what the respondent DOES or has DONE Ex; "how frequently"
opinion questions
what the respondent THINKS about something ex: "how much do you agree with.."
Under what circumstances are tables preferable to use compared to graphs?
when providing details and allow reference to specific values
