Chapter 2: Planing Evaluations for Health Education and Health Promotion Programs

Political Climate and Evaluation

- Conditions surrounding the political powers in an evaluation. - Interpretations of data will vary. Evaluators should attempt to identify the stakeholders for a health program. By identifying the motives of stakeholders, the evaluator can attempt to respond to wide range of concerns that will determine the usefulness of an evaluation.

Balance Between Evaluation Rigor and Utilization

- Consider two important perspectives on evaluation > evaluation rigor involves utilization of design methods including measurement, sampling, random assignment, and random selection to ensure internal and external validity OR evaluation utilization which focuses on the relevance of the evaluation to program decision makers (they could be concerned with real world constraints).

Conducting a Utilization-focused Evaluation

- Evaluation is focused on the needs of the consumer and practitioners. Focus on the needs of program decision makers and stakeholders. Goal is to provide useful information that is sensitive to political realties and program constraints. - This evaluation is designed around the information needs of program managers, program staff, and consumers. - This requires evaluator to engage decision makers in the evaluation planning process. 4 principles of Utilization-Focused Evaluation (42). - Evaluator will discuss potentials and limitations of evaluation data, defining the evidence needed to make program decisions, rigor of evaluation should match the importance and consequences of the decisions being made, and a method for implementation of decisions should be outlined. This will clarify the specific of how the evolution data will be interpreted and how the evaluation conclusions will be used by organizations to improve or refine the program.

implementation fidelity

- Extent to which the program was implemented as planned. - Important because it is a management tool (like accountability), foundation of evaluation plan (if the purpose of the evaluation was to determine the impact of a health program on health behavior or status, need to establish that the program was actually delivered.

Conducting a Rigorous Evaluation

- Garbage in, garbage out > garage in situation is when evaluation methods are not well-defined, rigorous, and reproducible which is likely to produce low quality evaluation. - Examples goals and objectives are not measurable, data gathering instruments do not establish validity and reliability, data gather instruments do not match goals and objectives, sampling methods, and alternative explanations are not addressed in the evaluation design - Must make evaluation methods rigorous. Stakeholders may debate the accuracy and interpretation of the results. The evaluator is the stakeholder that has to ensure the scientific credibility of the evaluation. - Steps to prevent garbage in of the evaluation: (1) Cleary define the purpose of the evaluation (2) create measurable program goals and objectives (3) Use rigorous scientific methods to produce high quality data (4) ensure that conclusions are grounded in high quality data. - This will help with ensuring a utilization focused evaluation that emphasizes close collaboration with stakeholders who will use the evaluation findings.

Cleary Define the Purpose of the Evaluation

- Need to come to a precise agreement on the purpose of the evaluation. Options can range from a evaluation that is limited to one stage of the logic model to a comprehensive evaluation that focuses on all stages of the health problem. - Issues: who will be using the evaluation results? what decisions will be made based on the evaluation results? - Allows the evaluator to propose an evaluation that is suited for the needs of the organization ad provides opportunity to clarify expectations about limitations of evaluation designs and the impact of these limits non quality of data and results.

comprehensive evaluation

- One that includes process, impact, and outcome evaluations methods. - Yields important insights into how health program produces results or fails to produce results. Not conducted often because they are expensive and takes years to conduct. - Need to decide on the scope and type of evaluation needed at a given time.

Health Program Planning and Evaluation: Integration

- Planning for the program and evaluation should begin at the same time and be linked to one another during the planning and implementing phases. When they are conducted simultaneously, they can be integrated with each other. - Three critical areas of information that evaluator needs to know before evaluating a health program: (1) Evaluator should learn the history of the program being evaluated (2) who will be actively involved in making decisions about the evaluation (informs the evaluator of the budget, time line, and purpose of the evaluation) (3) what is the purpose of the evaluation and what decisions will be made based n the results of the evaluation?

Logic Model and Comprehensive Evaluation

- Process evaluation: outputs, method used to assess degree to which program was implemented as designed. Procedures for measuring the extent to which participants received matched the program that was planned. Never assume that the program was implemented as designed because failing to document program implementation can affect stakeholder's interpretation of the impact and outcome evaluations. Gathers information about program quality and the satisfaction levels of participants, implementers, sites, and other personnel associated with the program. Quality assurance. - Impact evaluation: impact, focuses on participant changes in determinants of health behavior and determinants (includes forces outside the individual, which are the product of health promotion strategies). Targets health behaviors. These methods are used to determine the degree to which health education and promotion interventions caused changes in health behaviors or the determinants of health behaviors. (1) DETERMINANTS of health - knowledge, motivation, skills, health motivation, self-efficacy, perceived risk, and personal susceptibility - need to make sure to pay attention to measurement (2) HEALTH BEHAVIOR changes in the target population - logic model suggest that health program was delivered as designed, program produced intended enhancements in determinants of health behavior, and the determinants are sufficient to produce sustainable changes in health practices. Issue is that impact evaluation should sample the health behavior changes over a period of time. - Outcome evaluation: outcome, attainment of health status goals addressed. Concerned with assessing changes in health status of target population that are produced by changes in health practices - attention needs to be directed to the amount of time health practice needs to be adopted before changes in health status are detectable.

Political Climate and Program Stakeholders

- Program stakeholder is a person or organization having a keen interest in development, implementation, or evaluation of a health education promotion program. - Ex. funders, health organization administrators, professional personnel, target populations, general public, and evaluators. - Each stakeholder may look at the purpose of evaluation from a different perspective. - (1) Funders are interested in benefits, funds, resource use, presenting results to donors, replication, sustenance (2) organization implementing the health education or health promotion program: program impact, level of staff performance, material use, funds, cost, sustenance (3) members of the target population, worth of time and effort to participate, will program continue, pros and cons of the program, who will run the program, funds. (4) professional personnel (5) general public (6) evaluator - Different interests of stakeholders can create questioning of motives or conflict > personal meetings, formal coalition or advisory group of representative stakeholders, evaluation team that includes representatives from key stakeholders = shared decision making.

Ensure that Conclusions are Grounded in High Quality Data

- Report has to explain how evolution methods were employed, the quality of the data collected, data analysis methods and results. - Conclusions should be linked to data and limitations of evaluation methods should be described.

Use Rigorous Scientific Methods to Produce High Quality Data

- Rigor needs to be balanced with real world concerns like feasibility, cost, time, and health program disruption. Program staff are needed to develop a plan to employ evaluation rigor with reasonable cost and program disruption.

Create Measurable Program Goals and Objectives

- See if the goal and objective are measurable, is there a valid and reliable method of measuring each objective, do goals and objectives fall within the time frame of evaluation

logic model

- Systematic graphic representation of how an intervention is designed to produce it presets by showing its relationship to resources and activities. - Advantage is that it can be applied to wide range of program planning models in health education and promotion. - Systematic way to map out the relationship among resources for the program, activities, and intended results. Describes how objectives will be achieved. - Model should identify the goals and objectives of the program and activities of the program should be mapped out to explain how they are expected to achieve goals and objectives.

Evaluation and Continuous Health Program Improvement

- The Continuous health program improvement model is a cyclical model that starts off with health education knowledge base that informs the needs assessment, high quality program planning and implementation, and the program evaluation. - Health education knowledge base: Begins with professionals who are well-versed in health education knowledge base (this includes skills in application of systematic models of needs assessment, health behavior theory, health education program planning, and program evaluation; and includes understanding of linkages between determinants of health and health status of communities. - Needs assessment: Conduced to systematically identify the health needs and interest of the target community. Time to engage the target population in the process (consumer participation support and program ownership). Essential for resining health programs that are on target with participants interests. - Health program planning: Establishes health status goals, health behavior objectives, and strategies to achieve these goals. - Implementation: Evaluation procedures being. Pretesting done before the program begins and assessments are carefully administered to gather information information for evaluation. Post0testing and follow-up assessments are used to determine impact of health program on participants knowledge, skills, beliefs, attitudes, health behaviors, and health status. - Program evaluation: Reviewing goals and objectives (are the measurable, can they be refined). -Results of the program evaluation are used to refine and improve the health program. Evaluation results feed back into health education knowledge base, this enables practitioners to refine their decision making based on evidence of effective and ineffective practices.

Logic Model for Health Education and Health Promotion Programs

- Use these program planning models to systematically assess the health needs of the community and plan programs in response to those needs. (1) Planning Approach to community Health (PATCH) (2) PRECEDE/PROCEED model (3) Community Health Improvement Process (CHIP) model (4) Multilevel Approach to Community Health (MATCH) (5) RE-AIM (reach, effectiveness, adoption, implementation) - Common themes across health education and health promotion planning models: community participation (yields to many insights into both goals and objectives to achieve goals in a community), data-based decision making (can be used to inform decision making, set priorities, and target specific health concerns, behaviors, and resources), objectives that identify specific targets, activities to promote health actions (planning of health program and evaluation are occurring together)

impact

change in participants' knowledge, skills, motivation, and health behavior short term impact should be measurable from final session of the health program to 6 months after program exposure; long term impact are from 6 months to 5 years after program exposure impact evaluation

outcome

change in participants' or communities health status or quality of life produced by the health education/promotion program activities time depends on the relationship between health behavior change and changes in health status assessed 6 months to several years after health program implementation outcome evaluation

outputs

direct services that participants receive; these activities are designed to promote participant behavior capability and provide environmental supports to encourage health practices and enhance health status includes health education programs and environmental supports process evaluation

activities

educational processes, tools, events, and environmental supports planned to make up the program; this component designed to achieve objectives

inputs

human, financial, and community resources that are able to support the program; inputs are all resources needed to deliver health programs to target populations

team approach to program planning and evaluation

joint collaboration between those responsible for planning the program and those evaluating it advantages on page 36