Planning and Evaluating Exam 3
Financial Management
''Process of developing and using systems to ensure that funds are spent for the purpose for which they have been appropriated'' •Accounting, fiscal accountability•Audits - Internal and external
Quantitative data collection
(deductive; applying principle to case)-Produces numeric (hard) data, such as counts, ratings, scores-Suited to programs that are well defined
Justificiation
Degree of program approval by stakeholders
Support
Degree to which a support component is built into intervention
Recruitment
Degree to which members of priority population are adequately recruited through appropriate channels and places consistent with cultural characteristics
Multiplicity
Degree to which multiple components are built into program
Satisfaction
Degree to which needs of participants are being met, how satisfied they are with the program, service, or activity, and their belief that a positive impact is being made in their lives
Customer Orientation
Degree to which program is adapted to needs of population
Evidence
Degree to which program is evidence-based
Adjustment
Degree to which programs, services, or activities are modified based on feedback received from participants, partners, or other stakeholders
Market Analysis
identify competing behaviors, messages, programs
Qualitative Data Collection
inductive; examining case to form principle) -Produces narrative data, such as words and descriptions -Good for programs that emphasize individual outcomes or in cases where other descriptive information from participants is needed. -Useful during process evaluation.
Channel Analysis
communication preferences
Legal Concerns Associated with Implementation: Reducing the Risk of Liability
- reduce risk by planning ahead -Be aware of legal liabilities -Use certified instructors -Use good judgment -Require informed consent -Get medical clearance -Limit work to expertise -Provide a safe environment -Purchase insurance
Framework for Program Evaluation: Step 5 - Justifying Conclusions
-Compare evidence against standards of acceptability -Judging the worth, merit, or significance of program -Creating recommendations for actions based on results
Framework for Program Evaluation: Step 4 - Gathering Credible Evidence
-Decides on measurement indicators, sources of evidence, quality & quantity of evidence, & logistics for collecting evidence. -Organizes data including specific processes related to coding, filing, and cleaning.
Framework for Program Evaluation: Step 6 - Ensuring Use and Sharing Lessons Learned
-Disseminate or translate findings to real world -Needs of each group of stakeholders addressed
Procedures for Recordkeeping
-HIPAA-Records should be kept, but ensure anonymity / confidentiality -Credentialing
Competition
-Knowing what competition is offering helps in planning process -Identify competition during formative phase to understand consumer's perceived costs/benefits of competitors Ex: Elliot wants to lose weight before summer vacation. Competing Programs: Nutrisystem, Jenny Craig Competing messages: McDonald's, Wendy's
Framework for Program Evaluation: Step 3 - Focusing Evaluation Design
-Makes sure interests of stakeholders are addressed -Identifies reason of evaluation, how it will be used, questions to be asked, design of evaluation, & finalizes any agreements about the process.
Framework for Program Evaluation: Step 2 - Describing the Program
-Sets the frame of reference for all subsequent decisions in the evaluation process. -Describes mission, goals, objectives, capacity to effect change, stage of development, and how it fits into the larger community. -Logic model
Social Marketing Campaign Framework
1. Social Marketing Initiatives 2. Exposure to a 5 day message 3. Knowledge, attitudes and beliefs 4.Behavior
Standards of Evaluation
1. *Utility standards*-Ensure needs of users are satisfied 2. *Feasibility standards*-Ensure evaluation is viable and pragmatic 3. *Propriety standards*-Ensure evaluation is ethical 4. *Accuracy standards*-Ensure evaluation is correct
2 critical purposes of program evaluation:
1. Assessing & improving quality (Formative-Process evaluation 2. Determining program effectiveness(Summative)-Impact evaluation-Outcome evaluation
Planners should answer seven questions regarding price:
1. Who are the clients? 2. What is their ability to pay? 3. Are co-payers involved? 4. Is the program covered under an insurance program? 5. What is the mission of the planner's agency? 6. What are competitors charging? 7. What is the demand for the program or product?
Good logic models do several things:
1.Communicate purpose and expected results of program 2.Describe actions expected to lead to results 3.Become reference point for all involved 4.Improve program staff expertise in planning 5.Involve stakeholders, enhancing likelihood of funding 6.Incorporate findings from other projects 7.Identify potential obstacles to program operation
Create Procedural Manual and Participants' Manual
1.Ensures everyone understands program and its parameters 2.Standardize intervention so it can be replicated and avoid Type III errors (conclusions drawn not supported by evidence) 3.Provide ideas for facilitation 4.Provide additional background information on the topic5.Provide citations for additional resources.
3 basic components of a logic model:
1.Inputs (resources)-Resources to plan, implement, evaluate program •Human resources, partnerships, equipment, supplies, materials, and community resources 2.Outputs (activities or interventions)-Curricula, health services, infrastructure 3.Outcomes (results or effects)-Short-term, mid-term, and long-term
The following criteria can assist program planners in choosing appropriate segment
1.Measurable -How many people are in segment & can factors be measured? 2.Substantial -Is segment large enough & profitable enough to reach enough people to make a difference? 3.Accessible -Can segment be reached & services delivered?4.Differentiable -Are segments different enough that they will react differently to marketing strategies?5.Actionable -Can products be created to attract segments?
Phase 4: Putting Plans into Action
1.Pilot testing- test program with a small group from the priority population to identify any problems 2.Phasing in- limiting number of people who are exposed, then gradually increasing the numbers; by offerings, by location, by ability, by number 3.Total program- all in priority population are exposed at same time•Recommended each program use all 3 strategies but time and resources may not permit this First Day of Implementation-Program launch/Program kick off-Kick off in style!•Create a special event•Celebrity spokesperson-Promote!•Inform media representatives•Use press releases, video releases, spokespeople .•Consider releasing on symbolic day (First day of Spring) or time of year (New Year's Resolution)
Practical Problems or Barriers in Evaluation
1.Planners fail to build evaluation in planning process 2.Adequate resources not available to conduct appropriate evaluation. 3.Organizational restrictions prevent hiring consultants. 4.Effects are hard to detect because changes are small, come slowly, or do not last. 5.Length of time allotted for program and evaluation not realistic. 6.Restrictions limit collection of data among priority population. 7.Difficult to determine cause and effect. 8.Difficult to evaluate multi-strategy interventions. 9.Discrepancies between professional standards and actual practice exist 10.Evaluators' motives to demonstrate success introduce bias. 11.Stakeholders' perceptions of evaluation's value may vary too drastically. 12.Intervention strategies not delivered as intended or are not culturally specific.
Steps in segmenting for Health Promotion:
1.Review formative data to identify behaviors that influence health morbidity/mortality -Is behavior modifiable? -What is the potential impact of intervening? -What are current rates of behavior? -Is it possible to create program/product that will make it easy to change behavior? 2.Identify segmentation factors 3.Group people together who are similar
Purposes of Evaluating
1.To determine achievement of objectives 2.To improve program implementation 3.To provide accountability to funders, the community, and other stakeholders 4.To increase community support for initiatives 5.To contribute to scientific base for community public health interventions 6.To inform policy decisions
Resource
Adequacy of resources (e.g., budget, community resources, assets, time, etc.)
Technical Resources Management
All other resources besides human and financial •Equipment, expertise, information, material, partnerships
Concerns Associated with Implementation •Program Registration & Fee Collection
Establish payment procedures and a system of payment up front
Inclusion
Extent to which an adequate range & number of participants are involved in the program
Capacity
Extent to which professionals have adequate knowledge, skills, and abilities to design and implement program
Accountability
Extent to which the staff is fulfilling its responsibilities
Phase 1: Adoption of the Program
Part of marketing process designed to build initial interest in program
Standing Desk Intervention Study Examples of Logic Model
INPUTS: Research staffTrainingPartnership with CLASFunding and space from HHPDesksSurveillance cameras OUTPUTS: New knowledgePublicationsMedia Reports SHORT TERM OUTCOMES: Reduced sitting in class LONG TERM OUTCOMES: Improved health and academic performance of studentsUniversity wide adoption
Reporting and Documenting
Keep others informed about progress of program for: 1. accountability 2. public relations 3. motivation of present participants . 4. recruitment of new participants
Standards of Acceptability
Minimum levels of performance, effectiveness, or benefits used to judge the value of program
Types of Segment
Multiple factors can be used to segment; no right or wrong way to segment; can use primary (a priori) or secondary (a posteriori) data •Demographics (gender, age, etc.) •Benefits sought (weight loss, vanity) •Environment •Lifestyle/Psychographics (knowledge, attitudes, beliefs) •Behaviors (readiness to change)
Accelerating Diffusion of Innovation: Maloney's 16% Rule
Once you have reached 16% adoption of any innovation, you must change your messaging and media strategy from one based on scarcity to one based on social proof, in order to accelerate through the chasm to the tipping point. (between early adopters and early majority)
Market Mix
Planners need to make strategic decisions related to four marketing variables (the four Ps): Product: desired behavior change Price: costs to make the change Place: locations to deliver message Promotion: messages to encourage change
Evaluation
Process of determining the value or worth of a health promotion program or any of its components based on predetermined criteria or standards of acceptability identified by stakeholders.
Training for Facilitators
Program facilitators need to be familiar with the intervention •Either by participating in planning process or via training
Response
Proportion of population actually participating in program
Reach
Proportion of population given opportunity to participate in program
Diffusion Theory
Provides explanation for diffusion and patterns of adopting new innovations Planners can use DT to target marketing strategies, communication channel strategies and even intervention strategies
Interaction
Quality of interactions between planners & participants (e.g., customer service)
Approaches
Refers to formative, process, and summative evaluation and suggests these types of evaluation are clearly distinct.
Designs
Relates to summative evaluation; experimental, quasi-experimental, & non-experimental.
Consumer analysis
Wants, needs, preferences, barriers/facilitators to behavior or product use
Process Evaluation
a type of formative evaluation conducted to improve program •What was done, when, where, how, reach, engagement •Focuses on how a program achieves its goals •How closely did program follow implementation plan? •Identifying the critical ingredients of the program necessary for change
Logic Models visually convey connection between program
activities and desired outcomes Help stakeholders understand "Big picture'' of how planning, implementing, and evaluating all fit together •Increase likelihood program goals will be successful
Brand
an identifiable name, design or symbol that differentiates one group from another •First contact priority pop has with product •Impacts purchasing decision •Memorable
Consumer-based programs
are focused on the priority population's wants, needs, desires, preferences
Impact Evaluation
assesses program's effect on participants •Usually focus on intermediary or immediate measures such as awareness, knowledge, attitudes, behaviors, or skills
Outcome Evaluation
can focus on short and long-term goals. Ultimate Goal of program .•May demonstrate changes in health conditions, quality of life, and/or health behaviors •Mortality •Morbidity •Disability
Formative Evaluation
collects info that can be used primarily for ongoing program development and improvement •Provides info on achieving program goals or improving program •Usually collected before or duringimplementation but considered an ongoing process -Does not focus on outcomes
Promotion
communication strategy used to let priority population know about product, how to get it, and the benefits they will receive 1. Inform—increase product awareness or inform consumers 2. Persuade—convince people to purchase the product 3. Reinforce—remind them that the product exists 4. Differentiate—position the product as being different from the competition advertisements in newspapers, television, movie theatres, over urinals and in bars. Promotional items such as beer mats, beer can coolers and T-shirts were produced and distributed
Cost-identification analysis
compares different interventions available for a program to determine which intervention would be least expensive•What can we afford?
Social marketing
designing programs that facilitate voluntary behavior change for improved personal or societal well-being
Continuous Monitoring
determines program is operating at, above or below expectations. •Provides program planners with: -Data regarding level of program acceptance by priority population -Reach of messages -Product distribution sites -What is working and what is not working. --Improves efficiency and effectiveness of program by continually integrating feedback from the priority population.
Segmentation
divides priority population into smaller groups who share similar characteristics and who respond similarly to program •Helps narrow focus of marketing strategy •Increases chance of exchange •Improves efficacy of program =Targets true need
Cost-benefit analysis:
dollar benefit received from the dollars invested in the program.•Return on investment?
Success of any program measured by
effectiveness; quality of implementation
Pretesting
ensures planners have developed program components in response to, and reflective of the consumer's needs, wants, and expectations. •Allows for iteration (improving) product after first model has been developed •Tests hypothesis •2 phases of pretesting.
Legal Concerns Associated with Implementation: Negligence
failing to act in a prudent (reasonable) manner. Arises from two acts: -Omission - doing nothing when you should -Commission - doing something you should not
Phases of implementation
flexible and can be modified to meet different situations and circumstances faced by planners 1) adoption of the program 2) identifying and prioritizing the tasks to be completed 3) establishing a system of management 4) putting plans into action 5) ending or sustaining a program
Summative Evaluation
focuses on expected outcomes (effectiveness) stated in program goals and objectives •Did the intervention have an impact? •Includes both impact evaluation (intermediate outcomes) and outcome evaluation (long-term outcomes)
Elements of Formative Evaluation
justification, evidence, capacity, resources, customer orientation, multiplicity, support, inclusion, accountability, adjustment, recruitment, reach, response, interaction, satisfaction
Commercial Marketing
process of creating, communicating, and delivering value to customers
Phase 2 tests
promotional strategy messages/materials •Provides feedback and identifies red flags. •Common methods include focus groups and central location intercept interview •Process Evaluation •Who should complete pretesting? -Priority population for understanding -Experts in field for content validity -Gatekeepers for acceptance and approval Examples: •What about this message is/isn't motivating?•Where is the best place to reach you with this message?•What type of person is this message trying to reach?
Cost-effectiveness analysis
quantifies effects of program in monetary terms. •Money needed to have intended effect?
Product
what is being offered to meet customers' needs •Value to consumer •Accessible to consumer•"Augmented products" = product/service •Benefits to consumer = ''Core products'' or ''bundle of benefits'' -providing a ride to the bar, between bars and back home again
Price
what it costs priority population to obtain the product and its associated benefits •Barriers are what keep people from responding to an intervention or doing a behavior.•Benefits must outweigh costs •May be financial or nonfinancial -Social, mental, emotional, behavioral, psychological-Influences sense of quality of product around US$15 to US$20 for the evening; US$5 to US$10 for a single ride
Place
where the priority population has access to the product. •Location •Convenience •Safety •Time of day •Planners should make sure that it easy for the consumer to obtain the product or service home to bars and back again, so the customer's own vehicle would be left at home
Phase 1 Pre test
tests product concept •How likely they would be to use the product? •What they see as benefits to using the product? •What they see as barriers to using the product? •What product features they like?•What product features they would change, and why? •If the places selected to offer the product are convenient? •If the product price is reasonable?•If the benefits associated with product use are believable? •If the product functions as designed? •If instructions for how to use the product are clear?
Market
the set of all people who have an actual or potential interest in product or service
Implementation
to carry out or execute project plans; convert plans to action
Exchange
trading a product for certain costs In health promotion: -Seller = program planner -Consumer = priority population •Selling ideas, attitudes and behavior •Benefits of product must have value to consumer and be greater than what it costs to obtain product on their own
Consumer Orientation
understanding the needs and desires of your priority population Formative research answers questions of importance -What are they doing now to address this issue? -What are the benefits and limitations of current option? -What would make it easy for them to participate in new program? -What do they want out of this? -Where, when, how make it easiest to participate
Routes of promotion:
•Advertising •Direct marketing •Personal selling •Sales promotions •Public relations •Digital communication
Internal Evaluation
•An individual trained in evaluation and personally involved with the program conducts the evaluation. •Advantages -More familiar with organization and program history -Knows decision -making style of those in the organization -Present to remind people of results now and in the future -Able to communicate results more frequently and clearly -Less expensive •Disadvantages-Possibility of evaluator bias or conflict of interest
External Evaluation
•Conducted by someone not connected with the program •Advantages -More objective review and fresh perspective-Can ensure unbiased evaluation outcome-Brings global knowledge of working in a variety of settings-Typically brings more breadth and depth of technical expertise •Disadvantages-More expensive-Can be somewhat isolated, often lacking knowledge of and experience with the program •Evaluator should be credible, objective, have a clear role in evaluation design, and accurately report findings.
Key marketing principles that ensure success are same for marketing and social marketing:
•Continual focus on consumer's needs •Ensuring product meets consumer's needs and offers value •Ensuring product is delivered where the consumer is and when they want it •Ensuring product attracts consumers attention and prompts action
Safety Concerns Associated with Implementation
•Does program pose any potential safety concerns to population? •Informed consent-Purpose of program-Risks, dangers, discomfort -Expected benefits.-Inform of alternative programs. -They are free to discontinue participation at any time. -Allow participants to ask questions. •Do not protect planners from being sued. They just make participants aware of special concerns. •If the act of participating in program puts anyone at medical risk (e.g., cardiovascular exercise programs), individuals need to obtain medical clearance before participating. •Steps must be taken to ensure the safety and health of all associated with the program.-Safe location; appropriate security-Building codes met, facilities free of hazards-Qualified instructors-Plan in case of emergency
Evaluation in Program Planning Stages
•Evaluation design must reflect program goals and objectives •Evaluation must be planned in early stages and be in place before program begins. •Baseline data - measures reflecting initial status of participants •Initial data regarding program should be analyzed promptly to make any necessary adjustments •By creating summative evaluation early in the planning process, planners can ensure results are less biased.
Ethical Considerations
•Evaluation should never cause mental, emotional, or physical harm to those in priority population. •Participants should always be informed of the purpose & potential risks and should give consent. •No individual should ever have personal information revealed in any setting or circumstance. •When appropriate, evaluation plans should be approved by institutional review boards (IRBs).
Procedures Used in Formative Evaluation
•Focus groups •Surveys •In-depth interviews •Informal interviews •Key informant interviews •Expert panel reviews •Quality circles •Protocol checklist •Gantt chart •Program & evaluation forms •Direct observation
Phase 2: Identify & Prioritize Tasks to Be Completed
•Make a list of needs: 1. Hire/train staff 2. Reserve space 3. Post advertisements 4. Recruit and schedule participants 5. Implement intervention Program Evaluation & Review Technique (PERT) include diagram and timetable Critical Path Method (CPM) or PERT/CPM
Phase 3: Establish System of Management
•Management - process of assembling and using resources in a goal-directed manner to accomplish tasks in an organizational setting-human, financial, technical resources •The efficient, satisfactory management of a health promotion program is vital to its long-term success •Good management critical to ensure programs are: -Effective: meet stated goals and objectives -Efficient: well-organized, cost effective•Managers need varying skill sets: 1.Technical skills - specialized knowledge of procedures 2.Interpersonal skills - sensitive, persuasive, empathetic 3.Conceptual skills - logic, reasoning, judgment, analytics •4 Functions of Human Resources Management1.Planning - defining personnel (job descriptions, pay)2.Acquisition - hiring personnel (recruitment)3.Development - training personnel (onboarding)4.Sanction - maintaining personnel (discipline, rewards, safety)
Phase 5: Ending or Sustaining a Program
•Planners determine how long to run a program•If program meets goals, then program can end•Sustaining a program is more difficult.•5 techniques include: 1. Working to institutionalize 2. Seeking feedback from participants to demonstrate worth 3. Advocating for the program 4. Partnering with others 5. Revisiting & revising the rationale
Ethical Concerns Associated with Implementation
•Situations where competing values are at play & judgment must be made on what is most appropriate course of action •Code of Ethics for the Health Education Profession guides the work of health educators.-Planners should have integrity, be honest, loyal, accountable. •The Belmont Report: Ethical Principles & Guidelines for the Protection of Human Subject Research has 3 fundamental principles: 1.Respect for persons 2.Beneficence - maximizing benefits; doing good 3.Justice - fairness
Framework for Program Evaluation: Step 1 - ID and Engage Stakeholders
•Those involved in program operations •Those served or affected (directly or indirectly) by program •Primary users of the evaluation results
Virta Health Example
•What are they doing now to address this issue?-Insulin, drugs •What are the benefits and limitations of current option?-Keeps them alive but expensive and doesn't fix issue •What would make it easy for them to participate in new program?-Technology and health coaches that are accessible •What do they want out of this?-To be healthier, live enjoyable life •Where, when, how make it easiest to participate-Where they are, all day, app + health coaches
Evaluation Results
•Who will receive the results of the evaluation? •Different aspects of evaluation can be stressed, depending on group's needs and interests. •Different stakeholders may want different questions answered. •The planning for the evaluation should include a determination of how the results will be used.