SPEA V 379

Collecting Survey Data Disadvantages

Are more impersonal for participants, and usually not good for getting quotes in participants' own words

Rating Scale: Norm Referenced Scales

Asks for a comparison with some reference group in mind - Above average, below average, etc.

What strategies are used to decrease the costs of responding to a survey?

Avoid subordinating language Emphasize similarity to other requests

Limitations of Focus Groups

Less control over the course of the discussion Data are more complex to analyze Moderating requires training Large variability between groups Requires more effort to assemble More tailored environment required

Predictive Validity

The extent to which performance on one measure predicts performance on another

Concurrent Validity

The extent to which there are correlations among different measures of a construct.

Example of Ratio

Through the end of 2016 spring semester how many course credits had you earned at IU?

social exchange theory

the theory that our social behavior is an exchange process, the aim of which is to maximize benefits and minimize costs

survey is created that will only take 5 minutes to complete. Social Exchange Theory says

this will decrease social costs

purpose of collecting data on observations

to gather information on how the program operates, or on visual changes that have resulted.

Purpose of collecting Survey/questionnaires

to get information from the individual about their changes in knowledge, attitudes and behavior

Collecting Survey Data Advantages

valuation Can get information from a lot of people Can be done confidentially or anonymously, so may be more valid Can use questions from existing surveys More effective when using yes/no or true/false type questions

Nominal (categorical)

Objects cannot be ordered or added; they are labels.

Ordinal (ranked)

Objects of a set can be rank-ordered; equivalence between objects not guaranteed.

Types of Reliability

1) Instrument: extent to which the instruments give repeated results 2) Ratings/ Observation: extent of which different raters would code an observation the same way 3) Instrument Items: extent which related items on an instrument access the same construct 4) Construct Scales: extent which results are consistent across two different scales

Types of Validity

1) Statistical Conclusion: Establishes strength of the co-variation of cause and effect variable *2) Internal Validity: the extent to which an experiment shows convincingly that changes in behavior are a function of the independent variable and not the result of unknown variables.* 3) Measurement: how well do the measures capture the observable characteristics of a real world phenomena *4) External: extent which conclusion of the evaluation can be generalized to other time and places*

respondent background/demographics

CHARACTERISTICS of the respondent

Common Mistakes when writing survey questions

Choose simple words over specialized words Use complete sentences to ask questions Avoid vague quantifiers when more precise estimates can be provided Avoid specificity that exceeds an accurate answer Use equal numbers of positive and negative categories for scalar questions Avoid bias from unequal comparisons Don't use "all the apply" formats Develop response options that are mutually exclusive Avoid double barreled questions Avoid respondents to say yes in order to say no

Rating Scale: Criterion-Based Scales

Comparison against the respondent's absolute criteria for performance - Excellent, good, fair, poor, etc.

Collecting Interview Data Disadvantages

Data can be difficult and complex to analyze Interviewer can bias the responses.

Pie Charts

Dont show comparisons between data series well

what types of question are appropriate to appear as the first question on a survey instrument?

Easy to answer questions Should be relevant to the stated purpose of the survey

Systematic Inquiry

Evaluators conduct systematic and data-based inquiries Adhere to highest technical standards Explore strengths and shortcomings of evaluation questions and approaches Communicate approaches, methods and limitations accurately

Integrity/Honesty

Evaluators display honesty and integrity and attempt to ensure them throughout the entire evaluation process: Negotiate honestly with clients and stakeholders Disclose values, interests and conflicts of interest Represent accurately methods, data and findings Disclose source of request and financial support for evaluation

Competence

Evaluators provide competent performance to stakeholders: Possess appropriate skills and experience Demonstrate cultural competence Practice within limits of competence Continually improve competencies

Respect for people

Evaluators respect security, dignity and self-worth of ALL stakeholders. Understand evaluation context Get informed consent and protect confidentiality Maximize benefits and minimize harm Foster social equity Respect differences among stakeholders

General and public welfare

Evaluators take into account general and public interests Include relevant stakeholders Balance client and stakeholder needs Examine assumptions and potential side effects Present results in understandable forms

Collecting Interview Data Advantages

Excellent for both process and outcome evaluation, because you can get in-depth information from participants Can ask sensitive questions that require confidentiality

Collecting Data: Focus Groups Advantages

Good for outcome evaluation, because you can ask people to explain how the program affected them Can identify a lot of issues and effects Can give staff better understanding of the program from participants' own words

Collecting Data: Observational Groups Advantages

Good for process evaluation, because you can get an immediate impression about how the program is progressing

Focus Groups

Group of people (usually 7-10) Participants share some common characteristics Produce qualitative data - related to perceptions, attitudes of consumers about products, services, or opportunities (not intended to develop consensus or make decisions) Involve a focused discussion (carefully predetermined and sequenced)

Collecting Data: Observational Groups Disadvantages

Information can be difficult to interpret May be biased by how staff interpret the information Events that are seen can be time-dependent, and not representative of what is really going on If people know they are being observed, they may act differently than usual

Advantages of Focus Groups

Place people in real-life social situations (where they are influenced by other's opinions) Allow moderator to probe (or dig deeper) for information. Have high face validity Have relatively low costs Provide speedy results Allow for the inclusion of more respondents (compared to interviews)

Purpose of the Guiding principles

Promote ethical evaluation practice Foster continuing professional development Stimulate discussion within and outside evaluation Evaluators aspire to provide best possible information that might bear on the value of whatever is being evaluated

Informal Interviews Definition

Questions may vary between respondents, very dependent on interviewers conversational skills. large amount of flexibility

Interval

Rank-ordered scale in which the intervals represent equal distances; no absolute zero value.

Ratio

Rank-ordered scale with equal intervals and an absolute zero value.

Collecting Data: Focus Groups Disadvantages

Requires a good facilitator Takes time to analyze and interpret the discussion May require extra resources for facilitator's time and participant incentives

Implication of Survey Research

Reward Cost Trust

What elements are part of Social Exchange Theory

Reward, Trust and Cost

To assess knowledge, attitudes, and behaviors, which response options are most appropriate?

Strongly agree, agree, disagree and strongly disagree

What strategies are used to increase the rewards of responding to a survey?

Show positive regard Say thank you Support group values

Content Validity

Similar to face validity, but relies on experts in the field to determine the level of representation.

What strategies are used to increase trust among survey respondents?

Sponsorship from a legitimate authority Emphasize the task's important

Social Exchange Theory

The actions of individuals are motivated by the return they are expected to bring from others: ex: taking surveys

Instrument Reliability

The extent to which a measurement instrument gives the same reading if repeated in a given situation. EXAMPLE: A researcher is trying to measure eating habits among female collegiate athletes. During the pilot phase of the study, participants are recruited to complete a 24-hour food diary during week 3 of the semester. They also complete a 24-hour food diary during week 5 of the semester.

Inter-Rater Reliability

The extent to which different raters would code an observation or rating the same way. EXAMPLE: In order to determine if a classroom observation scale yields similar results for all members of the evaluation team, results are compared across a number of different observers.

Face Validity

The extent to which evaluation stakeholders agree the measure represents the construct (people who are looking at this generally agree that this is a good measure).

Validity vs Reliability

Validity: - does it do what it's supposed to do? does it work? Reliability - the degree to which the tool produces consistent/stable results *Tools can be valid and not reliable (and vice versa)

Example of Ordinal (ranked)

What is your current year in school 1.Fresh 2. Soph 3. Junior 4. Senior

Under what circumstances would standardized interviews offer more advantages than unstructured interviews?

When looking for basic information When asking a large number of people you want the questions to be the same to summarize the data easily. How many people are conducting the interviews What extent do you expect the results of the survey to fall into different categories?

How can data collected through focus groups compliment data collected through more quantitative data collection strategies such as surveys?

by helping identify thinking patterns and learn related vocabulary

Use Bar Charts when

comparing values, displaying quantitative values that relate to each other

Informal Interviews

conversational (unstructured)

response error

design the survey instrument to include well-written questions that are well-laid out. Clear and concise questions

standardized interview definition

everyone receives the same interview from a list of arranged questions. can be useful when interviewing a large number of people

knowledge questions

factual INFORMATION the respondent is familiar with ex: "are the following statements true or false"

Sampling Error

has to do with appropriate sample sizes

Sample Bias

has to do with representative samples

purpose of collecting data on focus groups

hold discussions with groups of 10-12 people go understand in-depth what they believe were the effects of the program, or how they saw the program implemented

focus groups can follow quantitative measures by

interpreting data that comes from a questionnaire, or to develop action strategies for concepts identified by the survey

Semi-structured

interview guider

Standardized Interview

interview protocol

Semi-structured interview definition

involves a list of questions that will be explored during an interview. fair amount of flexibility

Purpose of collecting Interviews

obtain information from individuals about their experiences, or to learn more about their answers to survey questions

what types of data are pie charts most likely to be helpful

part to whole data that relates parts of something to the whole

what type of information is best gathered through focus groups?

qualitative data

feeling questions

respondents EMOTIONAL response related to particular experiences or thoughts

use line charts when

showing relationship between variables and time

Example of Nominal

what is your major? _____

behavior questions

what the respondent DOES or has DONE Ex; "how frequently"

opinion questions

what the respondent THINKS about something ex: "how much do you agree with.."

Under what circumstances are tables preferable to use compared to graphs?

when providing details and allow reference to specific values