Myers
If the cost of delivery of a program was $50,000 and 1,000 clients were served, what would be the cost per client?
$50
The A in the SMART approach stands for ________.
Achievable
Which one of the following data displays would be the preferred method to compare two or more groups of people?
Bar chart
Which of the following is not a major content area for a program evaluation report?
Budget
Which of the following is not a best practice when coding qualitative data?
Code each relevant text to answer only one question
Which of the following is a disadvantage of using qualitative data collection methods?
Collecting and analyzing data can be expensive and time consuming
Which method below makes the effort to measure both costs and benefits in monetary units?
Cost-benefit
If a program evaluator wanted to determine which intervention to select among two possibilities based on cost, ________ would be the correct selection.
Cost-effectiveness
Culturally competent program evaluators typically and uniquely ________.
Demystify evaluations
Which one of the following is a desirable predisposition for an effective evaluator?
Discernment
A graphic representation of a program's key external memberships, resources, and competitors is called a(n) ________.
Ecomap
What is one of the broad types of qualitative analysis in program evaluation?
Ethnography
Identify the best definition of external validity [also known as generalizability] vs. internal validity.
Evaluation findings can be applied to groups or contexts beyond those being studied vs. being able to establish causal relationship between the program and the intended effect
Which one of these elements should be part of an effective evaluation plan?
Evaluator/evaluation team
What is an excellent source for measures of program outcomes?
Evidence-based instruments
What is the best way you can prepare a program for an evaluation?
Explore ways data collection can be integrated into the staff's daily functions
If an evaluator is addressing a highly controversial issue, what challenge do they face?
Facilitating different points of view
Identify which of the following is a possible pitfall during data collection.
Failing to account for drop-off in sample size due to attrition
Identify which of the following is a possible pitfall after data collection.
Failing to acknowledge the effects of multiple program components
What is the primary cause of managerial failure?
Faulty thinking rooted in inadequate ideas
Which one of these ideas is not one of the American Evaluation Association Guiding Principles?
Fidelity
Identify below the best example of a mixed methods evaluation design.
Focus groups and multiple choice surveys
Which one of the following is a characteristic of culturally responsive evaluation (CRE)?
Gives attention to groups that have been historically marginalized
The first stage of the evaluation process involves asking questions. Which one of the questions below would be least likely to be asked at this stage?
How do you limit the amount of data collected?
From the perspective of the Kellogg Foundation, what is considered the first of the planning steps in preparing for an evaluation?
Identifying stakeholders and establishing an evaluation team
The process of replacing missing data with substituted values is known as ________.
Imputation
In addition to the recommendation in question 3, what is another recommendation for conducting a client satisfaction survey?
Include one or two open-ended questions
What is one way to determine who the client system (target system) is?
Inclusion/exclusion admissions criteria
Which of the following is not a term synonymous with program theory?
Income premise
________ are markers of progress toward the change you hope to make with your strategy, initiative, or program.
Indicators
Which of the following possible project-level evaluation activities would fall within the maintenance and stability phase?
Inform alternative funding sources about accomplishments
How is a low reliability score interpreted?
Instrument is inconsistent
After designing an evaluation and collecting data, the information must be analyzed and ______.
Interpreted
A ________ organization is one in which the leadership and staff continually develop their ability to achieve their desired results through new and better ways of thinking and problem-solving.
Learning
An organization in which leadership and staff continually develop the ability to achieve the results desired is call a(n)____________.
Learning organization
Logic models are best described as ________ that should be used throughout the life of a strategy, initiative, or program.
Management tools
What is one caution in using agency information system data?
Missing data
Which of the choices below is one benefit among multiple benefits of developing and using a program logic model?
Ongoing evaluation of the program
Which of the following is a doable data analysis option for most program evaluators?
Percent change
In which type of evaluation method would an evaluator utilize quantitative data to check compliance with best practice/quality standards using expert or peer review?
Process evaluation
Which of these is not a potential resource for help in formulating the initial evaluation questions?
Professional Licensing Agencies
Which of the following is the best example of a program impact theory?
Program attempts to change people's attitudes and motivation toward alcohol in ways that provide them with the support necessary to avoid abuse
What is the key relationship in the cycle of events within a program?
Program data --> funding
What is one of the most important characteristics of an effective evaluation?
Provides usable information
Which threat to internal validity plays a potential role whenever control groups are used without random assignment?
Selection bias
What is one obstacle to reflecting on the findings and planning ways to use them identified in the Kellogg Step by Step Guide?
Staff not interested for various reasons
________ is one type of interview.
Structured
Maintenance inputs can be defined as the energetic imports that ________.
Sustain the system
What is the major problem with client satisfaction surveys?
Tendency for high ratings
How many evaluation questions are typically recommended?
Three to five
What is one criterion for evaluating a mission statement?
Timeless
What is one dilemma in prioritizing stakeholder input?
Who is left out
What is the most accurate definition of frame?
A mental model—a set of ideas and assumptions—that you carry in your head to help you understand and negotiate a particular territory
What can be defined as an organized collection of activities designed to reach certain objectives?
A program
Identify the main difference between an articulated program theory vs. an implicit program theory.
A program's theory is articulated when the program theory is spelled out in program documents and well understood by staff and stakeholders
A critical skill for a program evaluator is ________.
Ability to analyze a specific situation
All of the following are true regarding effective evaluation except ________.
An event that occurs at the end of a project
Developing and understanding a program's theory of change involves which one of these key factors?
Assumptions
Which of the following is not a use of the program evaluation findings?
Assuring compliance with all relevant regulations
Which of the following principles that help guide evaluation work says that evaluation should be concerned not only with specific outcomes, but also with the skills, knowledge, and perspectives acquired by the individuals who are involved with the project?
Build capacity
Program evaluations usually focus on determining objective ________ in the lives of clients, not on understanding subjective processes.
Changes
What type of need can be defined as an inferred measure of need by examining the characteristics of those receiving services and then locating those characteristics in the population?
Comparative need
One distinguishing difference between leaders that succeed at driving collaboration and innovation versus those that fail is their ability to grasp ________.
Complexity
Which of the following is not included on the checklist for selecting program evaluation instruments?
Comprehensiveness
Which of these paradigms focuses on understanding rather than explaining?
Constructivism
Case studies and narrative summaries about a particular piece of a program or participant are what type of techniques?
Contextualization analysis
What is the most important reason for conducting a program evaluation among the options provided below?
Continuous improvement of the program
A key consideration in planning an implementation evaluation is to ________.
Create a flexible and responsive design
An advantage of utilizing agency records is that the data is already available and does not have to be collected by the evaluator. However, there can also be drawbacks when using agency records for evaluation. Which of the following presents as a potential problem when utilizing agency records?
Data accuracy
What is the last step in needs assessments?
Formally disseminate results
Collect and utilize stakeholder input is Step ________ in the program evaluation process.
Four
What is one of the most powerful research tools available for evaluators to use in rigorously examining program outcomes?
Group research designs
An effective program goal statement will ________.
Harness staff and volunteer energy
Which of the following is a tip for conducting a focus group?
Have a process of managing group dynamics
What should you discuss during each planning and implementation step?
How you want to use the evaluation process to make action decisions
Culturally ________ evaluators will consider what client systems are excluded from the program evaluation process.
Humble
Using logic modeling and framing your evaluation questions, you can develop a ________.
Measurement framework
What is one guideline for program evaluation in a politically charged environment?
Negotiate a contract
What is one management consideration in determining and acquiring quantitative data sources?
Obtaining permission
Which one of these statements is not one of the criteria for selection of a data analysis approach?
Output specification
Which of the following is a basic unit of a logic model?
Outputs
What is one research method that is receiving increased utilization in developing countries and among many community-based initiatives?
Participatory evaluation
When engaging stakeholders in the evaluation process, there are several factors to take into consideration. Which of the following would not be a high priority consideration for engaging stakeholders?
Personality fit
There are typically three phases when developing a needs assessment: pre-assessment, assessment, and post-assessment. Which of the following is not a step of the pre-assessment phase?
Plan for data collection
When referring to the total existing number of cases in an area during a specified amount of time (i.e. total number of people in a specified county/city/state afflicted with influenza during a given month), which of the following would be the best term to utilize?
Prevalence
What is the key factor in determining what stakeholder input to highly value?
Purpose of the evaluation
A process evaluation may have three overarching purposes. Which one of these purposes is included?
Quality assurance
Not using words that make a judgment about people is an example of applying the principle of ________ in an evaluation.
Racial equity
The point of view that the world has an independent or objective existence apart from the perceptions of the observer is ________.
Realism
There are several types of stakeholders to consider when conducting an evaluation: implementers, partners, policymakers, program participants, and funders. Which of the following is the best example of a policymaker?
School board
Which measure will yield an indication of a client's confidence to be successful in a program?
Self-Efficacy Scale
Instruments should be ________.
Sensitive to change
Who is more likely to use evaluation if they understand and feel ownership over the evaluation process?
Staff, partners, and stakeholders
Which one of the following possibilities was mentioned in the Kellogg Handbook as a way to make your evaluation report more interesting?
Stage a debate or advocate-adversary analysis of the findings
What can best be defined as any person or group who has an interest in the project being evaluated or in the results of the evaluation?
Stakeholder
It is always possible to come across negative findings in program evaluations. Which of the following is not an appropriate method for reporting negative findings to stakeholders?
Start the report with negative findings and wrap up with positive findings at the end
One disadvantage of a focus group is that participants may ________.
Struggle with sharing opinions in a group
What is one issue in increasing the chances that an evaluation will be utilized?
Successfully dealing with negative findings
Which of the following open system functions most contributes to growth of the program?
Systems as cycles of events
Creating your own evaluation measurement tool is ________.
The last resort
What design involves providing an intervention or program to a group of clients and then determining the clients' status after intervention?
The one-group posttest-only design
A ________ is a narrative that explains the links between program strategies or activities and outcomes, and how and why the desired change is expected to come about.
Theory of change
Which of the following is one of the key considerations for developing a measurement framework?
Understand your capacity
When establishing a sampling frame, which of the following should be included?
Units that are members of the target population
One disadvantage of focusing on knowledge and attitudes in evaluation measures is that they may be ________.
Unrelated to behavior
What is one recommendation for use of client satisfaction surveys?
Use a "ballot box" approach
What is one ideal qualitative analysis approach?
Use of qualitative analysis software
When developing a communication plan for evaluation findings, evaluators should not ________.
Utilize technical jargon
Client satisfaction surveys can be thought of as ________.
Weather vanes
Which of the following is not a critical evaluation question to address over the life of a project?
What approach to logic modeling is appropriate for the project?
There are three major types of evaluation: performance monitoring, process or formative evaluation, and outcome or summative evaluation. Which of the following questions fits best with a process or formative evaluation?
What needs to be improved and how?