V379 Midterm 1
What are the limitations on the use of experimental designs?
often not feasible and hard to carry out
What is the difference between an outcome and an indicator?
outcome → change intended by the program's activitiesindicator → measurable statistic that tells us that change has happened must be specific, observable, measurable, and linked to outcome achievement ex) Improved reading capability → Standardized Test Scores
When is process evaluation appropriate?
when evaluating if program activities have been implemented in their intended way
4 Criteria for Good Evaluation Questions
1. Can we use data to answer the question? 2. Will your evaluation audiences care about the answer to the question? 3. Can your audiences use the answer to the question for further action? 4. Are the data available/accessible to answer the question? How is the program being delivered?
What type of questions does process evaluation address?
1. Is the program reaching the appropriate "target population" 2. Are the services delivered the program consistent with program design specifications?
What is included in the 5-Step Process for designing and administering surveys?
1. Overall design and planningpurpose of survey → what are you measuring and why who are respondentswhat kinds of questions 2. Selecting a methodconsider population, response quality, survey design, and response rate 3. Writing Survey QuestionsBehavior, opinion, feeling, knowledge, respondent questions 4. Data collection + Management pre survey notification letter 5. Data analysis and reporting
What are the two main types of quasi-experimental designs and what does each compare?
1. Pre Post Design: a number of observations are made in a natural context, before and after some target event compares Control and Intervention groups 2. Time Series Designs: variation of a "pre post" design, where a number of observations are made, in a natural context, before and after some target event compares participants in two different time periods
What are the three conditions that must be met to demonstrate that an intervention caused a specific outcome?
1. Temporal Asymmetry: the "causal variable" (the intervention) precedes the "effect variable" (the outcome) 2. Co-variation: as one variable changes (the application of the intervention), the other (the outcome) varies either positively of negatively MIDTERM STUDY GUIDE 4 3. Elimination of Plausible Rival Hypotheses: there are no other factors that could plausibly explain co-variation between the two variables
What are the different types of questions included in surveys?
Behavior Questions What the respondent does or has done? Response Scales: BehaviorsPer day, per week, per month, per year always, frequently, seldom, never always, sometimes, never Opinion Questions What the respondent thinks about something? Response ScalesCompletely satisfied, mostly satisfied, somewhat, dissatisfied Oppose, SupportStrongly disagree, disagree, agree, strongly agree Feeling Questions the respondent's emotional response related to particular experiences or thoughts Knowledge Questions the factual information the respondent is familiar with Response Scalesobjective (true or false, multiple choice), subjective Respondent Background/Demographics characteristics of the respondent
When comparison groups are used for quasi-experimental designs, what are the options for creating those groups?
Control groups as baseline (pre intervention characteristics) Intervention group usually randomly assigned, intended to be equivalent
What are the three primary types of economic evaluation? How do they differ from one another? In what cases might one model be more useful than the others?
Cost Effectiveness Analysis costs are monetized against a single non-monetized outcome combines information about program costs (dollars) with one specific outcome (units of effectiveness) → the result is a ratio of cost to units of effectiveness used with programs that have a single objective ex) cost per life saved, cost per person returning to employment, cost per illness prevented
What is the difference between criterion-based and norm-referenced scales?
Criterion Based Scales comparison against the respondent's absolute criteria for performance Excellent, good, fair, poor, etc Norm- Referenced Scales Asks for a comparison with some reference group in mind better/worse, above/below average
Rwanda's Tubeho Neza (Live Well) Program:
Implementation Elements → implemented through Rwanda ministry of health and social enterprise DelAgua Health to deliver environmental health tech Outputs → distribution of EcoZoom Dura improved wood burning cookstove and water filter a RCT was done to assess effects1/2 controls and 1/2 received interventions Radio ads Household Visits → CHW did first initial visit to deliver products and provide instruction additional visits for surveying Results → reduced prevalence of child diarrhea reduced acute respiratory infection
What are the various components of a logic model and what do they mean?
Inputs: resources invested that allow us to achieve the desired outputs program investments Outputs: activities conducted or products created that reach targeted participants or populations activities (what we do) + participation (who we reach) Outcomes: changes or benefits for individuals, families, groups, businesses, organizations, and communities short term (learning), intermediate (action and performance), long term (conditions)
Why are organizations typically most interested in process evaluation?
Intended to help organizations that are designed to help people or change social structures
What are the advantages/disadvantages of including a midpoint in a response scale?
Midpoint allows for measuring change when there might be a neutral opinion can be midpoint response bias
Five Main Types of Evaluation Questions
Needs Assessment Questions: questions about pre-existing conditions a program is intended to addressex) What type of food is most affordable and accessible in the bodegas where community members buy food? Assessment of Program Theory: questions about program conceptualization and designex) What types of skills are needed for Shop Healthy NYC staff to be successful? Process Evaluation Questions: questions about a program operations, service delivery, and participationex) What are the characteristics of bodega owners who are most enthusiastic about participating in Shop Healthy NYCex) How long are reading sessions? Outcome Evaluation Questions: questions about how the program changes attitudes, knowledge, behavior, and/or conditionsex) What types of food are purchased more frequently following the Shop Healthy NYC intervention? Efficiency (Economic) Assessment Questions: questions about program cost and cost effectivenessex) What is the public investment in Shop Healthy NYC, and does it lead to economic gains in other areas
What is the difference between outcome assessment and impact assessment?
Outcome assessment: Measurement of outcomes themselvesImpact Assessment: Measurement of the changes in outcomes that are attributable to the intervention itself
How are Outcomes different than Outputs?
Outputs are activities and participationthe activities that are implemented by the program participants: people involved people the program is targeted for Outcomes are short-term and intermediate short term (often focused on the learning part) intermediate ( often focused on action) behavior, decision making, social action
What are some of the reasons quasi-experimental designs are used for evaluation rather than experimental designs?
They usually involve real world interventions rather than experimental artificial laboratory settings
How to write consistent questions by using the root and a stem framework.
Root: How much/ How frequentlyStem (Question or Statement) Response Options: Strongly Disagree, Disagree, Agree, Strongly Agree
The Magnolia Mother's Trust:
Short term → virtual connection with other moms, job search info, personal improvement Intermediate → self efficacy, relocate out of subsidized housing, go back to school to continue education, increase in long term employment, more government public assistance program Participants → Black mothers with children 18 or younger aged 21-45 residing in 1 of the 4 STO subsidized housing communities in good standing with housing community What does program provide? $1000 monthly income for 12 monthsopportunity to open a 529 account for each minor children support of Community Specialistsfamily gatherings Agents of Change HIV Prevention Program: Purpose → AIDS is a prevalent condition in Africa 2/3 of all people living with HIV are found in Africa little HIV prevention - only in schools Evaluation Design → Quasi experimental study that compared non randomly chose intervention/ control groups any church could take part → church based peer education for sex education abstinence was not the only approachintervention groups → churches who were running the program control → churches not running the programself selection through church leaders selection bias can occur when groups are not chosen by random assignment Results can raise age of sexual debut + increase condom usehad no effect on # of sexual partners and secondary abstinence
10 tips for writing good survey questions
Simple words > specialized words Avoid leading words and questions avoid vague quantifiers when more precise estimates can be provided avoid specificity that messes up potential for response use equal numbers of positive and negative categories for scaled response avoid bias from unequal comparisons avoid check all that apply format use mutually exclusive response options avoid double barreled questions avoid asking respondents to say yes in order to say no
What are some examples of evaluation questions that can be answered by conducting economic evaluation?
What is the cost of the program or intervention? What is the economic burden of the condition? How do costs compare to outcomes?
Program Theory + Logic Models
a simplified picture of a program, initiative, policy, or intervention shows logical relationships among the resources that are invested, the activities that take place and the benefits or changes that result (program theory) helps differentiate between what is done (outputs) and results (outcomes) measures important variables
Cost Benefit Analysis
both costs and benefits to society are monetized to determine a net social benefit goes beyond CEA and CUA in monetizing the benefits of a program or intervention premise is that a "bottom line" for projects or programs can be assessed ex) how do the costs of providing a job training program for unemployed workers compare to the financial benefits compare to the financial benefits derived from the training?
Cost Utility Analysis (CUA)
costs are monetized against a non-monetized multiple outcome indicator Used to compare health and economic consequences of health and medical interventions multiple components and multiple outcomes ex) QALY (Quality-Adjusted Life Years) Index
Impact Evaluation: Quasi Experimental Designs
don't involve random assignment, but "matched groups" that are similar in critical ways most convincing option for determining program impact
What types of activities are examined in process evaluation?
focuses on service utilizationWho is using the program?To what extent are they engaging? Service delivery --- What is being delivered? How is this affected by other functions of the organization?
Public Opinion Polls - what is their purpose?
goal is to give everyone in the population (regardless of their wealth, age, education, race, knowledge) an equal voice about the issues of the day relies on random sampling - to be representative of a larger population
What are some of the challenges to using experimental designs for evaluation of programs and policies?
it involves significant technical and managerial challenges can be demanding in their technical and practical aspects participants may feel threatened by evaluation
Probability vs non-probability sampling
probability sampling: every member of the target population has a known nonzero chance of being selected for the sample - selection of sample is done randomly non probability sampling: a method of selecting units from a population using a subjective (i.e. non-random) method fast and inexpensive way of obtaining data
What is the difference between random sampling and random assignment?
random sample: all members of the study population have an equal chance of selection in the sample random assignment: a probabilistic procedure determines whether each individual (or other unit) in the evaluation sample will be a member of the intervention group or the control group also referred to as randomization
What are the primary limitations of each of these designs? (Hint:one involves bias, the other involves changes over time)
selection bias: the intervention and control groups differ (due to the selection method) in a way that influences the outcomes of the program time series → other changes over time can be attributed to the reason there was change in data
What is the goal of weighting polls and why is it done?
statistical process by which pollster adjusts poll data to ensure it represents target population overall adjusts for the fact that it is impossible to survey everyone in a large population, as well as the reality that the fraction of people polled may differ in certain important ways from the overall population whose opinions are sought without weighting, polls typically underrepresent younger, less educated, and non white adults who are less likely to respond to polls
What does outcome evaluation tell you about a program?
the condition of the target population that a program is expected to change knowledge, attitudes, behaviors, conditions measurement of outcomes
Why are experimental designs considered to be the "gold standard" of impact study designs?
they involve intervention and control groups, that are determined through random assignment intervention is delivered to a group that is equivalent to the control in all ways it provides the most convincing evidence of program impact
Against what types of standards can you judge program performance?
you can judge program performance by behavior change in targeted audience