Program Implementation and Evaluation

Steps in Evaluation

1.Involve the stakeholders 2.Describe the program 3.Focus the evaluation design on issues most important to the stakeholders 4.Gather credible evidence ("valid" & "reliable" evidence) 5.Justify conclusions Ensure use and share lessons learned (documentation)

questions to address during the implementation phase

1.Why: the effect of the objective to be achieved 2.What: the activities required to achieve the objective 3.Who: the individuals responsible for each activity 4.When: the chronologic sequence of activities 5.How: the materials, media, methods, and techniques to be used 6.How much: a cost estimate of materials and time

Pilot testing or pilot study what is it? what does it use and provide? what does it determine?

A method or small preliminary test to ensure a program or survey is usable Using a smaller population with the intent to expand later Provides an opportunity to test the program's effectiveness Determines if questions are interpreted as intended and that given answers include all possibilities Provides ease in control and monitoring of the program activities

qualitative vs quantitative- which is more important?

Both are equally important when evaluating a program. Each provides a unique perspective of the program.

qualitative evaluation

Descriptive, explanatory information Non-numerical data Observations, survey results, interviews Attitudes, opinions Helpful when long-term changes are expected More likely to tell us why something changed and what factors were involved Identify what people liked or did not like Lends to program improvement

questions to ask

Did the program accomplish what it was designed to do? Did the program work better than other similar programs? Did the program reduce health costs? How could the program be improved? Should the program be continued? Does the program merit continued funding? Should the program be expanded? How important was the problem toward which the program was directed? How much of the problem was solved? To what extent did the activities attain the objectives? What was the cost in resources to attain the objectives? What desirable and undesirable adverse affects occurred?

Evaluation focus

Did the program do what it was designed to do? Why did the program succeed or fail? §What unexpected positive or negative effects occurred? The MOST important focus is on improvement of processes, implementation, efficiency or anything that would make the program more organized and cost effective.

Sample: project homeless

Formative # volunteers needed # homeless clients served last year; age; ethnicity # agencies participating last year, what additional services needed Contacts with agencies Budget Timeline Summative # volunteers participating # homeless clients served: age, ethnicity... # agencies participating # and value of services provided Satisfaction of participating clients Satisfaction of participating agencies Budget Timeline Comparisons with reports from prior years

Evaluations- formative or Process

Helps point out problems & identify opportunities to make improvements Tied to routine operations with practical ongoing measurement of process and outcomes involving program staff and stakeholders

evaluation timing- when should it be done?

Ideally should be developed during the planning phase, PRIOR to implementation...NOT at the conclusion of the program! If this piece is not designed until the conclusion, the opportunity to collect pretest data is missed and the evaluation can be biased.

Quantitative evaluation

Identifies measures expressed as a quantity or amount, often on a numerical scale Objective and measurable information # of participants # surveys returned # of services provided # of volunteers Etc.

judging the value of a program

Merit or Quality Worth or Cost Effectiveness Significance or Importance A program may have merit but not worth the cost!

Intermediate outcomes

Might be a better measure for educational outcomes than long-term health outcomes May include variables such as changes in: -Knowledge -Attitudes -Behaviors Examples: -Intentions to quit smoking -Improvements in plaque levels -Decreased sugar consumption

reporting to stakeholders

No evaluation is worthwhile if the information is not reported back to the stakeholders Increases the relevance and credibility of the results Increases the likelihood of long-term participation User friendly reports with outcomes displayed graphically Recommendations for the futureshould be stated Remember the value of "thank you"to volunteers & donors

Summative Outcomes Post-program when and what to evaluate

Occurs AFTER the intervention is complete

Formative Process Pre-program when and what to evaluate

Occurs PRIOR to or DURING the implementation process

Information management systems

Organizes data to manage a program and make decisions Tracks information (numbers and types of service) Provides information for comparison Tracking forms Spreadsheets Provides information that will be tabulated, summarized and displayed graphically to make it useful and understandable

can pilot testing results be included in the evaluation results?

Pilot testing results should NOT be included in the evaluation results...WHY? Often pilot studies must be conducted before research funding can be obtained for complete studies

evaluation designs

Post-program only Pre-program and post-program Pre-program and post-program with a comparison group Pre-program and post-program with a control group

Purpose of evaluation

Provide a judgment of merit or worth about the program A means to developing good practice To make the best use of scarce resources To provide feedback to staff and participants To shape policy development Quality assurance to all interested parties Contribution to the provision of quality services to people in need

Evaluations- Summative or Outcome

Results of the program are compared with the goals and objectives Determines IMPACT Helps all interested parties make decisions about the value and possible continuance of programs

framework for program evaluation- steps (6), description, detail

Step Description Detail 1: Engage Stakeholders Persons or organizations having an investment in what will be learned and what will be done. Those involved in program operations, served or affected by the program and primary users of the evaluation. 2: Describe the Program Convey the mission & objectives of the program being evaluated. Statement of need, expected effects, activities, resources, state of development, context and logic model. 3: Focus the Evaluation Design The evaluation must assess the issues of greatest concern to stakeholders while using time and resources efficiently. Purpose, users, questions, methods, and agreements. 4: Gather Credible Evidence The information collected should convey a well rounded picture of the program that is credible. Indicators, sources, quality and logistics. 5: Justify Conclusions Conclusions are justified because they are based on evidence. Standards, analysis & synthesis, judgments and recommendations 6: Ensure Use & Share Lessons Learned Deliberate action is taken to make sure evaluation process and findings are used and disseminated. Design, preparation, feedback, follow-up, dissemination, and additional uses.

Data Collection Methods

Surveys -Self-administered -Telephone -Face-to-face Archival trend data Observation (indices and screenings) Record review Focus Groups Unstructured interview/narratives Open-ended questions on a written survey Participant observation Archival research (document analysis)

Implementation

The process of putting the plan into action Ongoing process and needs supervision and evaluation to ensure program effectiveness Involves individuals, agencies, and the community working together

Standards for "good" evaluation

Utility - a measure of relative satisfaction Feasibility - that which is achievable, project viability Propriety - appropriate, suitability Accuracy - free from error

Questions to ask when judging value of a program

What will be evaluated? What aspects of the program will be considered when judging a program's performance? What standards must be reached for the program to be considered successful? What evidence will be used to indicate how the program has performed? What conclusions regarding program performance are justified by comparing the available evidence to the selected standards? How will the lessons learned be used to improve public health effectiveness?

evaluation- it is NOT enough to:

summarize what went well and what did not -This is the problem with one day programs like GKS draw conclusions based on intuition The objectives themselves must be specifically addressed!