Program Development: Program Evaluation
Contingency perspective
- advocates use of different approaches based on: 1. The nature of the program being evaluated 2. The context of the evaluation 3. Information needs of stakeholders and intended use of results 4. Program resources
What are stakeholders?
- are individuals and groups affected by evaluation process and findings - help formulate questions for evaluation
Non-Experimental Designs
1. Cross-sectional designs 2. Cohort studies 3. Experimental design
Program Impact Evaluation
1. Focus primarily on the goals, objectives, and immediate effects related to a program 2. Evaluators employ the most rigorous methods for examining cause and effect relationship between the program and achievement of program goals and objectives -Summative purpose -Seek valid evidence concerning extent to which the program actually caused the outcomes
Program Evaluation:Appreciative Inquiry (AI) Approach
1. Focuses on organizational and program assets rather than identification of problems and deficits 2. Attempts to discover what is working well and envision a positive future (Best-case scenario) Most common method for conducting AI is 4D Model
Two broad categories of evaluation are what?
1. Formative evaluations 2. Summative evaluations
Creating an evaluation plan involves
1. Identifying stakeholders 2. Developing evaluation questions 3. Determining data needs 4 .Choosing evaluation methods and instruments
Utilization of Evaluation Results 4 types
1. Instrumental use 2. Conceptual use 3. Process use 4. Symbolic use
Data collection techniques include the following in Qualitative
1. Interviews 2. Focus groups 3. Observations 4. Review of documents
Formative Structure
1. Needs Assessment Evaluation 2. Program Theory Evaluation 3. Implementation Evaluation
Quantitative Designs includes what
1. Non-Experimental Designs 2. Quasi-Experimental Designs
Quasi-Experimental Designs
1. Nonequivalent control group design 2. Interrupted time-series design
What is logic model?
1. Once the problem is defined, programs develop theories 2. explain cause and effect links between problems, program activities, and outcomes
Summative Structure
1. Outcome Evaluation 2. Impact Evaluation 3. Efficiency Evaluation
Formative evaluations
1. Results center on program improvement 2. Provide information on program's theoretical framework, design, activities, and operations
Summative evaluations
1. Results center on sustainability of a program 2. Provide information on the program's outcomes, impact, and effectiveness 3. Provide evidence that changes experienced by the clients of a program result from the program, not other factors
Evaluators are different than researchers because
1. Stakeholders are part of the process; evaluators not fully in control 2. Context is critical component of the evaluation so generalization of results is limited
Participatory Approach
1. emphasizes the formative purposes of evaluation 2. Considers the information needs of all stakeholder groups rather than just the needs of the managers
Implementation evaluations
1. process evaluations, provide information on the operations of the program 2. Examine components of the program and the actual practices 3. Provide formative recommendations to ensure target population is being reached, activities are being offered, and program outcomes are being monitored
Program theory evaluation
1. provides a description of how a program is intended to work 2. Results indicate causal link between the activities and events of the program and intended outcomes 3. Often result in visual diagrams or logic models that represent causal links between elements and outcomes
Efficiency evaluations are used what?
1. to examine the cost and benefits of programs 2. Assess quality of operations 3. Outcomes they produce
Five principles for practice of AI approach
1.Constructivist principle 2.Simultaneity principle 3.Poetic principle 4.Anticipatory principle 5.Positive principle
Evaluations are generally conducted around five broad foci:
1.Needs assessment 2.Program theory 3.Program implementation 4.Program impact 5.Program efficiency
Experimental Designs
Randomized controlled trial
Evaluation research, also known as program evaluation, is used to make what?
a judgment of the value of a program
Process evaluation
also referred to as formative evaluation, assesses, analyzes, and documents the implementation of a program to insure that activities are conducted as planned
All evaluation approaches are concerned with the values, and merit or worth of programs, but what
differ in what elements of evaluation are emphasized
Objective Approach
emphasizes determination of achievement of stated goals and objectives
Implementation evaluations focus on what?
fidelity of program
Managerial Approach
includes examination of program objectives, but looks beyond outcomes, views informational needs of managers as a critical aspect of evaluation
Evaluators work with stakeholders to determine what?
main purpose of evaluation
Utilization-Focused Approach
not a specific design methodology, a process for including users in design of evaluation process and focuses on users' intended uses of the evaluation results
Process use
occurs when cognitive or behavior change occurs as a result of evaluation results
Planning evaluation should be done ______ to program implementation!
prior
Qualitative Designs
provide an in-depth perspective on phenomena that are not easily quantifiable
Utilization
refers to how people use the results of the evaluation
Evaluators seek to provide useful information to what?
stakeholder groups
Impact evaluations assess what?
the immediate effects of a program on the target population
Outcome evaluations focus on what
the long-term effects of a program
Needs assessments conducted to define what?
the problem/problems the program will address
To achieve intended outcomes, programs should be implemented as described in ________
theory
Conceptual use
use of evaluation data for political gain
Symbolic use
use of evaluation data for political gain
Instrumental use
using evaluation results to inform action
