Myers

If the cost of delivery of a program was $50,000 and 1,000 clients were served, what would be the cost per client?

$50

The A in the SMART approach stands for ________.

Achievable

Which one of the following data displays would be the preferred method to compare two or more groups of people?

Bar chart

Which of the following is not a major content area for a program evaluation report?

Budget

Which of the following is not a best practice when coding qualitative data?

Code each relevant text to answer only one question

Which of the following is a disadvantage of using qualitative data collection methods?

Collecting and analyzing data can be expensive and time consuming

Which method below makes the effort to measure both costs and benefits in monetary units?

Cost-benefit

If a program evaluator wanted to determine which intervention to select among two possibilities based on cost, ________ would be the correct selection.

Cost-effectiveness

Culturally competent program evaluators typically and uniquely ________.

Demystify evaluations

Which one of the following is a desirable predisposition for an effective evaluator?

Discernment

A graphic representation of a program's key external memberships, resources, and competitors is called a(n) ________.

Ecomap

What is one of the broad types of qualitative analysis in program evaluation?

Ethnography

Identify the best definition of external validity [also known as generalizability] vs. internal validity.

Evaluation findings can be applied to groups or contexts beyond those being studied vs. being able to establish causal relationship between the program and the intended effect

Which one of these elements should be part of an effective evaluation plan?

Evaluator/evaluation team

What is an excellent source for measures of program outcomes?

Evidence-based instruments

What is the best way you can prepare a program for an evaluation?

Explore ways data collection can be integrated into the staff's daily functions

If an evaluator is addressing a highly controversial issue, what challenge do they face?

Facilitating different points of view

Identify which of the following is a possible pitfall during data collection.

Failing to account for drop-off in sample size due to attrition

Identify which of the following is a possible pitfall after data collection.

Failing to acknowledge the effects of multiple program components

What is the primary cause of managerial failure?

Faulty thinking rooted in inadequate ideas

Which one of these ideas is not one of the American Evaluation Association Guiding Principles?

Fidelity

Identify below the best example of a mixed methods evaluation design.

Focus groups and multiple choice surveys

Which one of the following is a characteristic of culturally responsive evaluation (CRE)?

Gives attention to groups that have been historically marginalized

The first stage of the evaluation process involves asking questions. Which one of the questions below would be least likely to be asked at this stage?

How do you limit the amount of data collected?

From the perspective of the Kellogg Foundation, what is considered the first of the planning steps in preparing for an evaluation?

Identifying stakeholders and establishing an evaluation team

The process of replacing missing data with substituted values is known as ________.

Imputation

In addition to the recommendation in question 3, what is another recommendation for conducting a client satisfaction survey?

Include one or two open-ended questions

What is one way to determine who the client system (target system) is?

Inclusion/exclusion admissions criteria

Which of the following is not a term synonymous with program theory?

Income premise

________ are markers of progress toward the change you hope to make with your strategy, initiative, or program.

Indicators

Which of the following possible project-level evaluation activities would fall within the maintenance and stability phase?

Inform alternative funding sources about accomplishments

How is a low reliability score interpreted?

Instrument is inconsistent

After designing an evaluation and collecting data, the information must be analyzed and ______.

Interpreted

A ________ organization is one in which the leadership and staff continually develop their ability to achieve their desired results through new and better ways of thinking and problem-solving.

Learning

An organization in which leadership and staff continually develop the ability to achieve the results desired is call a(n)____________.

Learning organization

Logic models are best described as ________ that should be used throughout the life of a strategy, initiative, or program.

Management tools

What is one caution in using agency information system data?

Missing data

Which of the choices below is one benefit among multiple benefits of developing and using a program logic model?

Ongoing evaluation of the program

Which of the following is a doable data analysis option for most program evaluators?

Percent change

In which type of evaluation method would an evaluator utilize quantitative data to check compliance with best practice/quality standards using expert or peer review?

Process evaluation

Which of these is not a potential resource for help in formulating the initial evaluation questions?

Professional Licensing Agencies

Which of the following is the best example of a program impact theory?

Program attempts to change people's attitudes and motivation toward alcohol in ways that provide them with the support necessary to avoid abuse

What is the key relationship in the cycle of events within a program?

Program data --> funding

What is one of the most important characteristics of an effective evaluation?

Provides usable information

Which threat to internal validity plays a potential role whenever control groups are used without random assignment?

Selection bias

What is one obstacle to reflecting on the findings and planning ways to use them identified in the Kellogg Step by Step Guide?

Staff not interested for various reasons

________ is one type of interview.

Structured

Maintenance inputs can be defined as the energetic imports that ________.

Sustain the system

What is the major problem with client satisfaction surveys?

Tendency for high ratings

How many evaluation questions are typically recommended?

Three to five

What is one criterion for evaluating a mission statement?

Timeless

What is one dilemma in prioritizing stakeholder input?

Who is left out

What is the most accurate definition of frame?

A mental model—a set of ideas and assumptions—that you carry in your head to help you understand and negotiate a particular territory

What can be defined as an organized collection of activities designed to reach certain objectives?

A program

Identify the main difference between an articulated program theory vs. an implicit program theory.

A program's theory is articulated when the program theory is spelled out in program documents and well understood by staff and stakeholders

A critical skill for a program evaluator is ________.

Ability to analyze a specific situation

All of the following are true regarding effective evaluation except ________.

An event that occurs at the end of a project

Developing and understanding a program's theory of change involves which one of these key factors?

Assumptions

Which of the following is not a use of the program evaluation findings?

Assuring compliance with all relevant regulations

Which of the following principles that help guide evaluation work says that evaluation should be concerned not only with specific outcomes, but also with the skills, knowledge, and perspectives acquired by the individuals who are involved with the project?

Build capacity

Program evaluations usually focus on determining objective ________ in the lives of clients, not on understanding subjective processes.

Changes

What type of need can be defined as an inferred measure of need by examining the characteristics of those receiving services and then locating those characteristics in the population?

Comparative need

One distinguishing difference between leaders that succeed at driving collaboration and innovation versus those that fail is their ability to grasp ________.

Complexity

Which of the following is not included on the checklist for selecting program evaluation instruments?

Comprehensiveness

Which of these paradigms focuses on understanding rather than explaining?

Constructivism

Case studies and narrative summaries about a particular piece of a program or participant are what type of techniques?

Contextualization analysis

What is the most important reason for conducting a program evaluation among the options provided below?

Continuous improvement of the program

A key consideration in planning an implementation evaluation is to ________.

Create a flexible and responsive design

An advantage of utilizing agency records is that the data is already available and does not have to be collected by the evaluator. However, there can also be drawbacks when using agency records for evaluation. Which of the following presents as a potential problem when utilizing agency records?

Data accuracy

What is the last step in needs assessments?

Formally disseminate results

Collect and utilize stakeholder input is Step ________ in the program evaluation process.

Four

What is one of the most powerful research tools available for evaluators to use in rigorously examining program outcomes?

Group research designs

An effective program goal statement will ________.

Harness staff and volunteer energy

Which of the following is a tip for conducting a focus group?

Have a process of managing group dynamics

What should you discuss during each planning and implementation step?

How you want to use the evaluation process to make action decisions

Culturally ________ evaluators will consider what client systems are excluded from the program evaluation process.

Humble

Using logic modeling and framing your evaluation questions, you can develop a ________.

Measurement framework

What is one guideline for program evaluation in a politically charged environment?

Negotiate a contract

What is one management consideration in determining and acquiring quantitative data sources?

Obtaining permission

Which one of these statements is not one of the criteria for selection of a data analysis approach?

Output specification

Which of the following is a basic unit of a logic model?

Outputs

What is one research method that is receiving increased utilization in developing countries and among many community-based initiatives?

Participatory evaluation

When engaging stakeholders in the evaluation process, there are several factors to take into consideration. Which of the following would not be a high priority consideration for engaging stakeholders?

Personality fit

There are typically three phases when developing a needs assessment: pre-assessment, assessment, and post-assessment. Which of the following is not a step of the pre-assessment phase?

Plan for data collection

When referring to the total existing number of cases in an area during a specified amount of time (i.e. total number of people in a specified county/city/state afflicted with influenza during a given month), which of the following would be the best term to utilize?

Prevalence

What is the key factor in determining what stakeholder input to highly value?

Purpose of the evaluation

A process evaluation may have three overarching purposes. Which one of these purposes is included?

Quality assurance

Not using words that make a judgment about people is an example of applying the principle of ________ in an evaluation.

Racial equity

The point of view that the world has an independent or objective existence apart from the perceptions of the observer is ________.

Realism

There are several types of stakeholders to consider when conducting an evaluation: implementers, partners, policymakers, program participants, and funders. Which of the following is the best example of a policymaker?

School board

Which measure will yield an indication of a client's confidence to be successful in a program?

Self-Efficacy Scale

Instruments should be ________.

Sensitive to change

Who is more likely to use evaluation if they understand and feel ownership over the evaluation process?

Staff, partners, and stakeholders

Which one of the following possibilities was mentioned in the Kellogg Handbook as a way to make your evaluation report more interesting?

Stage a debate or advocate-adversary analysis of the findings

What can best be defined as any person or group who has an interest in the project being evaluated or in the results of the evaluation?

Stakeholder

It is always possible to come across negative findings in program evaluations. Which of the following is not an appropriate method for reporting negative findings to stakeholders?

Start the report with negative findings and wrap up with positive findings at the end

One disadvantage of a focus group is that participants may ________.

Struggle with sharing opinions in a group

What is one issue in increasing the chances that an evaluation will be utilized?

Successfully dealing with negative findings

Which of the following open system functions most contributes to growth of the program?

Systems as cycles of events

Creating your own evaluation measurement tool is ________.

The last resort

What design involves providing an intervention or program to a group of clients and then determining the clients' status after intervention?

The one-group posttest-only design

A ________ is a narrative that explains the links between program strategies or activities and outcomes, and how and why the desired change is expected to come about.

Theory of change

Which of the following is one of the key considerations for developing a measurement framework?

Understand your capacity

When establishing a sampling frame, which of the following should be included?

Units that are members of the target population

One disadvantage of focusing on knowledge and attitudes in evaluation measures is that they may be ________.

Unrelated to behavior

What is one recommendation for use of client satisfaction surveys?

Use a "ballot box" approach

What is one ideal qualitative analysis approach?

Use of qualitative analysis software

When developing a communication plan for evaluation findings, evaluators should not ________.

Utilize technical jargon

Client satisfaction surveys can be thought of as ________.

Weather vanes

Which of the following is not a critical evaluation question to address over the life of a project?

What approach to logic modeling is appropriate for the project?

There are three major types of evaluation: performance monitoring, process or formative evaluation, and outcome or summative evaluation. Which of the following questions fits best with a process or formative evaluation?

What needs to be improved and how?