Planning and Evaluating Exam 3

Financial Management

''Process of developing and using systems to ensure that funds are spent for the purpose for which they have been appropriated'' •Accounting, fiscal accountability•Audits - Internal and external

Quantitative data collection

(deductive; applying principle to case)-Produces numeric (hard) data, such as counts, ratings, scores-Suited to programs that are well defined

Justificiation

Degree of program approval by stakeholders

Support

Degree to which a support component is built into intervention

Recruitment

Degree to which members of priority population are adequately recruited through appropriate channels and places consistent with cultural characteristics

Multiplicity

Degree to which multiple components are built into program

Satisfaction

Degree to which needs of participants are being met, how satisfied they are with the program, service, or activity, and their belief that a positive impact is being made in their lives

Customer Orientation

Degree to which program is adapted to needs of population

Evidence

Degree to which program is evidence-based

Adjustment

Degree to which programs, services, or activities are modified based on feedback received from participants, partners, or other stakeholders

Market Analysis

identify competing behaviors, messages, programs

Qualitative Data Collection

inductive; examining case to form principle) -Produces narrative data, such as words and descriptions -Good for programs that emphasize individual outcomes or in cases where other descriptive information from participants is needed. -Useful during process evaluation.

Channel Analysis

communication preferences

Legal Concerns Associated with Implementation: Reducing the Risk of Liability

- reduce risk by planning ahead -Be aware of legal liabilities -Use certified instructors -Use good judgment -Require informed consent -Get medical clearance -Limit work to expertise -Provide a safe environment -Purchase insurance

Framework for Program Evaluation: Step 5 - Justifying Conclusions

-Compare evidence against standards of acceptability -Judging the worth, merit, or significance of program -Creating recommendations for actions based on results

Framework for Program Evaluation: Step 4 - Gathering Credible Evidence

-Decides on measurement indicators, sources of evidence, quality & quantity of evidence, & logistics for collecting evidence. -Organizes data including specific processes related to coding, filing, and cleaning.

Framework for Program Evaluation: Step 6 - Ensuring Use and Sharing Lessons Learned

-Disseminate or translate findings to real world -Needs of each group of stakeholders addressed

Procedures for Recordkeeping

-HIPAA-Records should be kept, but ensure anonymity / confidentiality -Credentialing

Competition

-Knowing what competition is offering helps in planning process -Identify competition during formative phase to understand consumer's perceived costs/benefits of competitors Ex: Elliot wants to lose weight before summer vacation. Competing Programs: Nutrisystem, Jenny Craig Competing messages: McDonald's, Wendy's

Framework for Program Evaluation: Step 3 - Focusing Evaluation Design

-Makes sure interests of stakeholders are addressed -Identifies reason of evaluation, how it will be used, questions to be asked, design of evaluation, & finalizes any agreements about the process.

Framework for Program Evaluation: Step 2 - Describing the Program

-Sets the frame of reference for all subsequent decisions in the evaluation process. -Describes mission, goals, objectives, capacity to effect change, stage of development, and how it fits into the larger community. -Logic model

Social Marketing Campaign Framework

1. Social Marketing Initiatives 2. Exposure to a 5 day message 3. Knowledge, attitudes and beliefs 4.Behavior

Standards of Evaluation

1. *Utility standards*-Ensure needs of users are satisfied 2. *Feasibility standards*-Ensure evaluation is viable and pragmatic 3. *Propriety standards*-Ensure evaluation is ethical 4. *Accuracy standards*-Ensure evaluation is correct

2 critical purposes of program evaluation:

1. Assessing & improving quality (Formative-Process evaluation 2. Determining program effectiveness(Summative)-Impact evaluation-Outcome evaluation

Planners should answer seven questions regarding price:

1. Who are the clients? 2. What is their ability to pay? 3. Are co-payers involved? 4. Is the program covered under an insurance program? 5. What is the mission of the planner's agency? 6. What are competitors charging? 7. What is the demand for the program or product?

Good logic models do several things:

1.Communicate purpose and expected results of program 2.Describe actions expected to lead to results 3.Become reference point for all involved 4.Improve program staff expertise in planning 5.Involve stakeholders, enhancing likelihood of funding 6.Incorporate findings from other projects 7.Identify potential obstacles to program operation

Create Procedural Manual and Participants' Manual

1.Ensures everyone understands program and its parameters 2.Standardize intervention so it can be replicated and avoid Type III errors (conclusions drawn not supported by evidence) 3.Provide ideas for facilitation 4.Provide additional background information on the topic5.Provide citations for additional resources.

3 basic components of a logic model:

1.Inputs (resources)-Resources to plan, implement, evaluate program •Human resources, partnerships, equipment, supplies, materials, and community resources 2.Outputs (activities or interventions)-Curricula, health services, infrastructure 3.Outcomes (results or effects)-Short-term, mid-term, and long-term

The following criteria can assist program planners in choosing appropriate segment

1.Measurable -How many people are in segment & can factors be measured? 2.Substantial -Is segment large enough & profitable enough to reach enough people to make a difference? 3.Accessible -Can segment be reached & services delivered?4.Differentiable -Are segments different enough that they will react differently to marketing strategies?5.Actionable -Can products be created to attract segments?

Phase 4: Putting Plans into Action

1.Pilot testing- test program with a small group from the priority population to identify any problems 2.Phasing in- limiting number of people who are exposed, then gradually increasing the numbers; by offerings, by location, by ability, by number 3.Total program- all in priority population are exposed at same time•Recommended each program use all 3 strategies but time and resources may not permit this First Day of Implementation-Program launch/Program kick off-Kick off in style!•Create a special event•Celebrity spokesperson-Promote!•Inform media representatives•Use press releases, video releases, spokespeople .•Consider releasing on symbolic day (First day of Spring) or time of year (New Year's Resolution)

Practical Problems or Barriers in Evaluation

1.Planners fail to build evaluation in planning process 2.Adequate resources not available to conduct appropriate evaluation. 3.Organizational restrictions prevent hiring consultants. 4.Effects are hard to detect because changes are small, come slowly, or do not last. 5.Length of time allotted for program and evaluation not realistic. 6.Restrictions limit collection of data among priority population. 7.Difficult to determine cause and effect. 8.Difficult to evaluate multi-strategy interventions. 9.Discrepancies between professional standards and actual practice exist 10.Evaluators' motives to demonstrate success introduce bias. 11.Stakeholders' perceptions of evaluation's value may vary too drastically. 12.Intervention strategies not delivered as intended or are not culturally specific.

Steps in segmenting for Health Promotion:

1.Review formative data to identify behaviors that influence health morbidity/mortality -Is behavior modifiable? -What is the potential impact of intervening? -What are current rates of behavior? -Is it possible to create program/product that will make it easy to change behavior? 2.Identify segmentation factors 3.Group people together who are similar

Purposes of Evaluating

1.To determine achievement of objectives 2.To improve program implementation 3.To provide accountability to funders, the community, and other stakeholders 4.To increase community support for initiatives 5.To contribute to scientific base for community public health interventions 6.To inform policy decisions

Resource

Adequacy of resources (e.g., budget, community resources, assets, time, etc.)

Technical Resources Management

All other resources besides human and financial •Equipment, expertise, information, material, partnerships

Concerns Associated with Implementation •Program Registration & Fee Collection

Establish payment procedures and a system of payment up front

Inclusion

Extent to which an adequate range & number of participants are involved in the program

Capacity

Extent to which professionals have adequate knowledge, skills, and abilities to design and implement program

Accountability

Extent to which the staff is fulfilling its responsibilities

Phase 1: Adoption of the Program

Part of marketing process designed to build initial interest in program

Standing Desk Intervention Study Examples of Logic Model

INPUTS: Research staffTrainingPartnership with CLASFunding and space from HHPDesksSurveillance cameras OUTPUTS: New knowledgePublicationsMedia Reports SHORT TERM OUTCOMES: Reduced sitting in class LONG TERM OUTCOMES: Improved health and academic performance of studentsUniversity wide adoption

Reporting and Documenting

Keep others informed about progress of program for: 1. accountability 2. public relations 3. motivation of present participants . 4. recruitment of new participants

Standards of Acceptability

Minimum levels of performance, effectiveness, or benefits used to judge the value of program

Types of Segment

Multiple factors can be used to segment; no right or wrong way to segment; can use primary (a priori) or secondary (a posteriori) data •Demographics (gender, age, etc.) •Benefits sought (weight loss, vanity) •Environment •Lifestyle/Psychographics (knowledge, attitudes, beliefs) •Behaviors (readiness to change)

Accelerating Diffusion of Innovation: Maloney's 16% Rule

Once you have reached 16% adoption of any innovation, you must change your messaging and media strategy from one based on scarcity to one based on social proof, in order to accelerate through the chasm to the tipping point. (between early adopters and early majority)

Market Mix

Planners need to make strategic decisions related to four marketing variables (the four Ps): Product: desired behavior change Price: costs to make the change Place: locations to deliver message Promotion: messages to encourage change

Evaluation

Process of determining the value or worth of a health promotion program or any of its components based on predetermined criteria or standards of acceptability identified by stakeholders.

Training for Facilitators

Program facilitators need to be familiar with the intervention •Either by participating in planning process or via training

Response

Proportion of population actually participating in program

Reach

Proportion of population given opportunity to participate in program

Diffusion Theory

Provides explanation for diffusion and patterns of adopting new innovations Planners can use DT to target marketing strategies, communication channel strategies and even intervention strategies

Interaction

Quality of interactions between planners & participants (e.g., customer service)

Approaches

Refers to formative, process, and summative evaluation and suggests these types of evaluation are clearly distinct.

Designs

Relates to summative evaluation; experimental, quasi-experimental, & non-experimental.

Consumer analysis

Wants, needs, preferences, barriers/facilitators to behavior or product use

Process Evaluation

a type of formative evaluation conducted to improve program •What was done, when, where, how, reach, engagement •Focuses on how a program achieves its goals •How closely did program follow implementation plan? •Identifying the critical ingredients of the program necessary for change

Logic Models visually convey connection between program

activities and desired outcomes Help stakeholders understand "Big picture'' of how planning, implementing, and evaluating all fit together •Increase likelihood program goals will be successful

Brand

an identifiable name, design or symbol that differentiates one group from another •First contact priority pop has with product •Impacts purchasing decision •Memorable

Consumer-based programs

are focused on the priority population's wants, needs, desires, preferences

Impact Evaluation

assesses program's effect on participants •Usually focus on intermediary or immediate measures such as awareness, knowledge, attitudes, behaviors, or skills

Outcome Evaluation

can focus on short and long-term goals. Ultimate Goal of program .•May demonstrate changes in health conditions, quality of life, and/or health behaviors •Mortality •Morbidity •Disability

Formative Evaluation

collects info that can be used primarily for ongoing program development and improvement •Provides info on achieving program goals or improving program •Usually collected before or duringimplementation but considered an ongoing process -Does not focus on outcomes

Promotion

communication strategy used to let priority population know about product, how to get it, and the benefits they will receive 1. Inform—increase product awareness or inform consumers 2. Persuade—convince people to purchase the product 3. Reinforce—remind them that the product exists 4. Differentiate—position the product as being different from the competition advertisements in newspapers, television, movie theatres, over urinals and in bars. Promotional items such as beer mats, beer can coolers and T-shirts were produced and distributed

Cost-identification analysis

compares different interventions available for a program to determine which intervention would be least expensive•What can we afford?

Social marketing

designing programs that facilitate voluntary behavior change for improved personal or societal well-being

Continuous Monitoring

determines program is operating at, above or below expectations. •Provides program planners with: -Data regarding level of program acceptance by priority population -Reach of messages -Product distribution sites -What is working and what is not working. --Improves efficiency and effectiveness of program by continually integrating feedback from the priority population.

Segmentation

divides priority population into smaller groups who share similar characteristics and who respond similarly to program •Helps narrow focus of marketing strategy •Increases chance of exchange •Improves efficacy of program =Targets true need

Cost-benefit analysis:

dollar benefit received from the dollars invested in the program.•Return on investment?

Success of any program measured by

effectiveness; quality of implementation

Pretesting

ensures planners have developed program components in response to, and reflective of the consumer's needs, wants, and expectations. •Allows for iteration (improving) product after first model has been developed •Tests hypothesis •2 phases of pretesting.

Legal Concerns Associated with Implementation: Negligence

failing to act in a prudent (reasonable) manner. Arises from two acts: -Omission - doing nothing when you should -Commission - doing something you should not

Phases of implementation

flexible and can be modified to meet different situations and circumstances faced by planners 1) adoption of the program 2) identifying and prioritizing the tasks to be completed 3) establishing a system of management 4) putting plans into action 5) ending or sustaining a program

Summative Evaluation

focuses on expected outcomes (effectiveness) stated in program goals and objectives •Did the intervention have an impact? •Includes both impact evaluation (intermediate outcomes) and outcome evaluation (long-term outcomes)

Elements of Formative Evaluation

justification, evidence, capacity, resources, customer orientation, multiplicity, support, inclusion, accountability, adjustment, recruitment, reach, response, interaction, satisfaction

Commercial Marketing

process of creating, communicating, and delivering value to customers

Phase 2 tests

promotional strategy messages/materials •Provides feedback and identifies red flags. •Common methods include focus groups and central location intercept interview •Process Evaluation •Who should complete pretesting? -Priority population for understanding -Experts in field for content validity -Gatekeepers for acceptance and approval Examples: •What about this message is/isn't motivating?•Where is the best place to reach you with this message?•What type of person is this message trying to reach?

Cost-effectiveness analysis

quantifies effects of program in monetary terms. •Money needed to have intended effect?

Product

what is being offered to meet customers' needs •Value to consumer •Accessible to consumer•"Augmented products" = product/service •Benefits to consumer = ''Core products'' or ''bundle of benefits'' -providing a ride to the bar, between bars and back home again

Price

what it costs priority population to obtain the product and its associated benefits •Barriers are what keep people from responding to an intervention or doing a behavior.•Benefits must outweigh costs •May be financial or nonfinancial -Social, mental, emotional, behavioral, psychological-Influences sense of quality of product around US$15 to US$20 for the evening; US$5 to US$10 for a single ride

Place

where the priority population has access to the product. •Location •Convenience •Safety •Time of day •Planners should make sure that it easy for the consumer to obtain the product or service home to bars and back again, so the customer's own vehicle would be left at home

Phase 1 Pre test

tests product concept •How likely they would be to use the product? •What they see as benefits to using the product? •What they see as barriers to using the product? •What product features they like?•What product features they would change, and why? •If the places selected to offer the product are convenient? •If the product price is reasonable?•If the benefits associated with product use are believable? •If the product functions as designed? •If instructions for how to use the product are clear?

Market

the set of all people who have an actual or potential interest in product or service

Implementation

to carry out or execute project plans; convert plans to action

Exchange

trading a product for certain costs In health promotion: -Seller = program planner -Consumer = priority population •Selling ideas, attitudes and behavior •Benefits of product must have value to consumer and be greater than what it costs to obtain product on their own

Consumer Orientation

understanding the needs and desires of your priority population Formative research answers questions of importance -What are they doing now to address this issue? -What are the benefits and limitations of current option? -What would make it easy for them to participate in new program? -What do they want out of this? -Where, when, how make it easiest to participate

Routes of promotion:

•Advertising •Direct marketing •Personal selling •Sales promotions •Public relations •Digital communication

Internal Evaluation

•An individual trained in evaluation and personally involved with the program conducts the evaluation. •Advantages -More familiar with organization and program history -Knows decision -making style of those in the organization -Present to remind people of results now and in the future -Able to communicate results more frequently and clearly -Less expensive •Disadvantages-Possibility of evaluator bias or conflict of interest

External Evaluation

•Conducted by someone not connected with the program •Advantages -More objective review and fresh perspective-Can ensure unbiased evaluation outcome-Brings global knowledge of working in a variety of settings-Typically brings more breadth and depth of technical expertise •Disadvantages-More expensive-Can be somewhat isolated, often lacking knowledge of and experience with the program •Evaluator should be credible, objective, have a clear role in evaluation design, and accurately report findings.

Key marketing principles that ensure success are same for marketing and social marketing:

•Continual focus on consumer's needs •Ensuring product meets consumer's needs and offers value •Ensuring product is delivered where the consumer is and when they want it •Ensuring product attracts consumers attention and prompts action

Safety Concerns Associated with Implementation

•Does program pose any potential safety concerns to population? •Informed consent-Purpose of program-Risks, dangers, discomfort -Expected benefits.-Inform of alternative programs. -They are free to discontinue participation at any time. -Allow participants to ask questions. •Do not protect planners from being sued. They just make participants aware of special concerns. •If the act of participating in program puts anyone at medical risk (e.g., cardiovascular exercise programs), individuals need to obtain medical clearance before participating. •Steps must be taken to ensure the safety and health of all associated with the program.-Safe location; appropriate security-Building codes met, facilities free of hazards-Qualified instructors-Plan in case of emergency

Evaluation in Program Planning Stages

•Evaluation design must reflect program goals and objectives •Evaluation must be planned in early stages and be in place before program begins. •Baseline data - measures reflecting initial status of participants •Initial data regarding program should be analyzed promptly to make any necessary adjustments •By creating summative evaluation early in the planning process, planners can ensure results are less biased.

Ethical Considerations

•Evaluation should never cause mental, emotional, or physical harm to those in priority population. •Participants should always be informed of the purpose & potential risks and should give consent. •No individual should ever have personal information revealed in any setting or circumstance. •When appropriate, evaluation plans should be approved by institutional review boards (IRBs).

Procedures Used in Formative Evaluation

•Focus groups •Surveys •In-depth interviews •Informal interviews •Key informant interviews •Expert panel reviews •Quality circles •Protocol checklist •Gantt chart •Program & evaluation forms •Direct observation

Phase 2: Identify & Prioritize Tasks to Be Completed

•Make a list of needs: 1. Hire/train staff 2. Reserve space 3. Post advertisements 4. Recruit and schedule participants 5. Implement intervention Program Evaluation & Review Technique (PERT) include diagram and timetable Critical Path Method (CPM) or PERT/CPM

Phase 3: Establish System of Management

•Management - process of assembling and using resources in a goal-directed manner to accomplish tasks in an organizational setting-human, financial, technical resources •The efficient, satisfactory management of a health promotion program is vital to its long-term success •Good management critical to ensure programs are: -Effective: meet stated goals and objectives -Efficient: well-organized, cost effective•Managers need varying skill sets: 1.Technical skills - specialized knowledge of procedures 2.Interpersonal skills - sensitive, persuasive, empathetic 3.Conceptual skills - logic, reasoning, judgment, analytics •4 Functions of Human Resources Management1.Planning - defining personnel (job descriptions, pay)2.Acquisition - hiring personnel (recruitment)3.Development - training personnel (onboarding)4.Sanction - maintaining personnel (discipline, rewards, safety)

Phase 5: Ending or Sustaining a Program

•Planners determine how long to run a program•If program meets goals, then program can end•Sustaining a program is more difficult.•5 techniques include: 1. Working to institutionalize 2. Seeking feedback from participants to demonstrate worth 3. Advocating for the program 4. Partnering with others 5. Revisiting & revising the rationale

Ethical Concerns Associated with Implementation

•Situations where competing values are at play & judgment must be made on what is most appropriate course of action •Code of Ethics for the Health Education Profession guides the work of health educators.-Planners should have integrity, be honest, loyal, accountable. •The Belmont Report: Ethical Principles & Guidelines for the Protection of Human Subject Research has 3 fundamental principles: 1.Respect for persons 2.Beneficence - maximizing benefits; doing good 3.Justice - fairness

Framework for Program Evaluation: Step 1 - ID and Engage Stakeholders

•Those involved in program operations •Those served or affected (directly or indirectly) by program •Primary users of the evaluation results

Virta Health Example

•What are they doing now to address this issue?-Insulin, drugs •What are the benefits and limitations of current option?-Keeps them alive but expensive and doesn't fix issue •What would make it easy for them to participate in new program?-Technology and health coaches that are accessible •What do they want out of this?-To be healthier, live enjoyable life •Where, when, how make it easiest to participate-Where they are, all day, app + health coaches

Evaluation Results

•Who will receive the results of the evaluation? •Different aspects of evaluation can be stressed, depending on group's needs and interests. •Different stakeholders may want different questions answered. •The planning for the evaluation should include a determination of how the results will be used.