Chapter 12 - Evaluation Research

Name three things Evaluation and basic research have in common

1) Both use the entire range of data gathering processes 2) Both can focus on cause and effect 3) Both face methodological issues with sampling and measurement

What kinds of things need to be determined in order to know if program evaluation is possible/feasible?

1) Determine purpose for evaluation. 2) Study and gather information on the program. 3) Develop a flow model of the program. 4) Identify program aspects that are sufficiently unambiguous in terms of inputs, goals, and linkages that evaluation of them is feasible.

Name 4 Barriers to Evaluation Research

1) Evaluation may not produce clear cut results 2) Poor communication between evaluators 3) Failure of evaluators to advocate for use 4) Resistance to change

Contrast evaluation research vs. basic research

1) Evaluation: Practical application; Basic: General information and hypothesis testing. 2) Evaluation: shaped by needs of sponsors; General: shaped by researcher interest 3) Evaluation: Judgmental; Basic: Neutrality 4) Evaluation: adapted to program; Basic: Rigor of research method (NOIR) 5) Evaluation: Results retained by sponsor; Basic: Publication and dissemination

Name two types of program evaluation

1) Formative 2) Summative

Discuss what is involved in program planning and implementation (First goal of formative program evaluation).

1) Gather information on target population, Existing services, Intervention strategies, Requisite skills, Project feasibility 2) May take form of needs assessment 3) May involve pretest or trial run

What are three issues when creating cost-benefit analysis?

1) How to estimate and monetize costs 2) Benefits and costs do not simultaneously accrue 3) Who bears the costs, and who reaps the benefits? Not always the same set of people.

Discuss what is involved in program monitoring (Second goal of formative program evaluation). What questions does this goal answer?

1) Is the target population in fact being served? 2) Are the services that are supposed to be delivered actually being delivered? 3) Is the quality of the service adequate?

Who are the stakeholders in program evaluation? (Three)

1) Program funders 2) Program administrators/personnel 3) Program clients/beneficiaries

Discuss what is involved in summative program evaluation and what its purpose/goals are.

1) Summative program evaluation involves assessing the impact of a program. 2) Intended to inform policy-making decisions: a) Whether to continue, expand, or cancel a program. b) Whether to generalize the findings of this project to other settings and populations.

What are two things you can do to aid in your variable measurement for program analysis?

1) Use multiple indicators where possible 2) Use an existing measure

What are some issues facing minority populations in evaluation research?

1) Variables tend to be selected based on stereotypes 2) Failure to get a representative sample 3) Performance evaluations are subject to political manipulation, which leaves minorities vulnerable

Name 3 reasons for why we evaluate

1. Administrative purposes (documentation for funding, program quality control) 2. Impact assessment (identity program goals, determine how well program achieves goals) 3. Test hypothesis or evaluate practice (add to professional's knowledge base)

Name two purposes for Evaluation research

1. Program Design and Development 2. Program Effect and Outcome

Define evaluation research.

A means of supplying valid and reliable evidence regarding the operation of social policies, programs and other types of social interventions regarding: how they are planned, how well they operate, and how effectively they achieve their goals.

What is cost effective analysis?

Compares program cost (measured in dollars) to program effects (measured in whatever units are appropriate), with program outcomes the program is intended to produce. For example, compared to the expense of the program, does it reduce juvenile arrests by a significant amount?

What are some nonpositivist critiques of evaluation?

Evaluation is beholden to multiple stakeholders, and therefore multiple perspectives are possible. Social interventions are subject to social meaning and interpretations. These interventions often reflect and reinforce inequitable distributions of power and resources in society.

How do you produce a cost-benefit analysis?

First calculate the dollar cost of the program and subtract that from the dollar value benefits of the program. This produces either a net gain (benefits > costs) or a net loss (benefits < costs).

What is formative program evaluation?

Formative program evaluation focuses on providing information to guide the planning, development, and implementation of a specific program.

What is summative program evaluation?

Summative program evaluation focuses on effectiveness and efficiency of programs and extent to which the outcomes can be generalized.

Within the context of measuring a variable for program analysis, define goal.

The desired end states for a program.

Give an example of how program goals are linked to specific programs using the topic of Teen Pregnancy. What would the independent and dependent variables be?

The indepent variable is the program and the dependent variable is the goal. Sex ed counseling, relationship counseling, and self-esteem enhancement can be used to impact the goal of delayed pregnancy. Self-esteem enhancement, as well as tutoring services and child care services also serve the goal of continuing education for teen mothers. Parenting classes would impact the goal of increasing mother-child bond.

Within the context of measuring a variable for program analysis, define objective.

The measurable criteria for success.

What is an opportunity cost?

The value of a lost/missed/forgone opportunity.

True or false? Evaluation and the programs they focus on are political in nature.

True.