Chapter 6 Training Evaluation
One-time costs
Choice Costs related to needs assessment and program development
Which of the following training outcomes is used to ascertain the extent to which trainees know the facts, procedures, techniques, principles, and processes taught in the training program?
Cognitive outcomes
Identify the methods that are used to minimize the threats to the validity of a company's training program evaluation. (Check all that apply.)
Comparison groups pretest and post-tests random assignment
The Evaluation Process
Conduct a Needs Analysis Develop Measurable learning objectives and analyze transfer of training. Develop Outcome Measures Choose an evaluation strategy Plan and execute the evaluation
Costs per trainee
Costs related to meals, materials, and lost productivity incurred to replace trainees who are attending the training program
Learning or cognitive outcomes
Directly measure the extent to which trainees have mastered the training content
True or false: A reliable test allows the trainer to have confidence that any improvements in post-training test scores from pretraining levels are the result of test characteristics or the test environment.
False Reason: This is false. A reliable test allows the trainer to have confidence that any improvements in post-training test scores from pretraining levels are the result of learning that occurred in the training program, not test characteristics or the test environment.
True or false: Pilot testing is used for summative evaluation rather than formative evaluation.
False Reason: This is false. Pilot testing should be used for formative evaluation. It can be used as a dress rehearsal to show the program to managers, trainees, and customers.
True or false: Self-assessments rather than tests and quizzes are the preferred measures to assess cognitive outcomes.
False Reason: This is false. Tests and quizzes rather than self-assessments are the preferred measures of learning. This is because self-assessments are only moderately related to learning and influenced by how much learners liked the course or were motivated to learn rather than what they actually learned.
True or false: The use of workforce analytics is not necessary to use big data.
False Reason: This is false. Using big data requires the use of workforce analytics.
Identify an accurate statement about the resource requirements model.
It compares personnel, equipment, facilities, and materials costs across various stages of training.
Which of the following statements is true of the Solomon four-group evaluation design used for the evaluation of a company's training program?
It controls for most threats to internal and external validity.
Identify a characteristic of the pretest/post-test training evaluation design.
It fails to eliminate the effects of business conditions or other factors that affect the behavior of trainees.
Which of the following statements is true about the formative evaluation of a training program?
It gives information about the ways to make the training program better.
Identify an accurate statement about return on investment (ROI), which is measured for training and development programs.
It helps in ascertaining the economic value provided by these programs.
Internal validity
The believability of the study results.
Practicality
The convenience with which the training outcome measures can be accumulated
Criterion contamination
The degree that training outcomes are influenced by extraneous conditions or evaluate unsuitable capabilities
An evaluation of training that is conducted to ascertain the extent to which trainees have changed after participating in a training program is known as ______.
summative evaluation
The practice of using scientific and quantitative methods to examine data from various data sources to reach evidence-based decisions and show the effect of human resource practices on important company metrics is referred to as _____.
workforce analytics
Brian is a new employee at New Design Technologies, an interior design company. He underwent a three-month employee training and has been working for two months since the completion of the training. If his manager wants to analyze whether training Brian has generated economic value for the company, the manager should _____.
find out the profit gained by the company as a result of the clients attracted by Brian
Return on investment (ROI) analysis is suitable for employee training programs that are _____.
focused on an operational issue
The evaluation of training that takes place during the designing and development phase of a training program is called _____.
formative evaluation
In the context of Kirkpatrick's five-level framework for categorizing training outcomes, level 3 outcomes are collected to determine _____.
if the newly learned skills are being implemented on the job
In an organizational context, arrange the steps involved in the training evaluation process in the correct order of occurrence. (Place the first step at the top.)
1. Conducting a needs analysis 2. Developing measurable learning objectives and analyzing of training 3. Developing outcome measures 4. Selecting an eval strategy 5. Planning and executing evaluation
Arrange the steps involved in the process of determining the return on investment (ROI) of an employee training program in the correct order of occurrence. (Place the first step at the top.)
1. Understanding the objectives of the training program 2. Developing plans for collecting data related to the training objectives 3. Isolating the effects of the program from other factors that might affect the data related to the training objectives 4. Converting the data related to the training objectives to a monetary value and calculating ROI
What do skill-based training outcomes include? (Check all that apply.)
Acquisition of skills Application of skills on the job
___ _____ refers to complex data sets developed by compiling data across different organizational systems.
Big Data
Indirect Costs
General office supplies and administrative and staff support salaries.
The _____ refers to participants in an evaluation study performing at a high level simply because of the attention they are receiving and not due to the training.
Hawthorne Effect
Identify the true statements about random assignment, a method used to minimize the threats to the validity of a company's training program evaluation. (Check all that apply.)
It helps reduce the differences between the training group and comparison group in ability, knowledge, skill, or other personal characteristics. It helps ensure that the trainees are similar in individual characteristics such as age, gender, ability, and motivation.
Which of the following is true of the accounting method of determining costs involved in training programs?
It identifies when the costs are incurred.
Which of the following is true of summative evaluation?
It includes estimating an organization's monetary benefits from a training program.
Which of the following statements is true about the pretest/post-test with comparison group design for evaluating training?
It requires collection of pretraining and post-training outcome measures from trainee and comparison groups.
According to Kirkpatrick's five-level framework for categorizing training outcomes, which of the following levels of outcomes are collected at the completion of training, before trainees return to the job?
Level 2 outcomes Level 1 outcomes
Identify the levels of Kirkpatrick's framework that are related to affective training outcomes, depending on how they are evaluated.
Level 4 Level 3 Level 2
Pretraining measure
One way to improve the internal validity of the study results is is to first establish a baseline called...
Identify the training evaluation design that controls for most threats to validity.
Pretest/post-test with comparison group
Direct costs
Program material and supplies and salaries for all employees involved in training.
Reaction outcomes
Provide information regarding the extent to which the trainer, facilities, or learning environment may have hindered learning
Which of the following training evaluation outcomes is level 1 criteria in Kirkpatrick's five-level framework?
Reaction outcomes
Self assessments
Refers to a learners estimates of how much they know or have learned from training.
Summative evaluation
Refers to an evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program.
Return on investment
Refers to comparing the trainings monetary benefits with the cost of training.
Threats to validity
Refers to factors that will lead an evaluator to question either (1) the believability of the study results or (2) the extent to which the evaluation results are generalizable to other groups of trainees and situations
Training outcomes/criteria
Refers to measures that the trainer and the company use to evaluate training programs.
Training effectiveness
Refers to the benefits that the company and the trainees receive from training.
Evaluation design
Refers to the collection of information, including what, when, how, and from whom -- That will be used to determine the effectiveness of the training program.
Formative evaluation
Refers to the evaluation of training that takes place during program design and development.
Criteria relevance
Refers to the extent to which training outcomes are related to the learned capabilities emphasized in the training program.
Training evaluation
Refers to the process of collecting the outcomes needed to determine whether training is effective.
Pilot testing
Refers to the process of previewing the training program with potential trainees and managers or with other customers (persons who are paying for the development of the program).
Which of the following training outcomes involves comparing the monetary benefits of a training with its cost?
Return on investment
Which of the following training outcomes needs to be determined in order justify funding and to increase the status of a training and development function?
Return on investment
_____ refer to learners' estimates of how much they know or have learned from training.
Self-assessments
__________________ outcomes are used to assess the level of technical or motor skills and behaviors of trainees.
Skill-based
Identify the ways that determine the value of training using a design that can rule out and isolate its effects on results.
Success cases Return on expectations (ROE)
Discrimination
The extent to which the performance of trainees on the outcome reflects differences in performance on the job
Reliability
The extent to which training outcomes can be evaluated consistently over a time period
Criterion deficiency
The failure to evaluate training outcomes that were emphasized in the training objectives
Variety
The large number of data sources and different types of available data
Volume
The large quantity of available data
Velocity
The speed with which the large amount of generated data must be evaluated, captured, and made useful
Benefits
The value that the company gains from the training program.
Skill-based outcomes
These are used to assess the level of technical or motor skills and behaviors.
Cognitive outcomes
These are used to determine the degree to which trainees are familiar with the principles, facts, techniques, procedures, and processes emphasized in the training program.
Identify an accurate statement about affective training outcomes.
They involve trainees' motivation and attitudes.
Identify an accurate statement about reaction outcomes of a training program.
They refer to trainees' perceptions of the training program.
Affective outcomes
This definition includes attitudes and motivation.
Why is the training cost information important?
To compare the costs involved in other similar training programs To determine total expenses on training To determine the proportion of money spent on administration, training development, and evaluation
External Validity
Trainers interested in the generalizability of the study results to other groups and situations.
Costs Per Offering
Training site rental fees and trainer salaries
True or false: Reaction and cognitive outcomes of a company's training program do not help assess how much the trainees actually use the training content in their jobs.
True Reason: This is true. Reaction outcomes of a company's training program provide information regarding the extent to which the trainer, facilities, or learning environment may have hindered learning. Learning or cognitive outcomes directly measure the extent to which the trainees have mastered the training content. However, reaction and cognitive outcomes do not help determine how much the trainees actually use the training content in their jobs.
Identify the methods that tend to be helpful in identifying the benefits of employee training. (Check all that apply.)
Use of practitioner, technical, and academic literature Estimates of training benefits provided by trainees and their managers
Results
Used to determine the training programs payoff for the company.
Identify the cost-benefit analysis method that involves determining the dollar value of training on the basis of different factors such as estimates of the difference in job performance between trained and untrained employees and the number of individuals trained.
Utility analysis
Utility analysis uses _____ to estimate the difference in job performance for trained versus untrained employees.
a pretest/post-test design with a comparison group
Results outcomes are used to ascertain _____.
a training program's payoff for an organization
Hundred new employees of HFCT Bank completed the bank's employee training program. Harold, the HR manager, wants to know if the employees liked the training program. In this scenario, Harold should _____.
ask them to take a survey that includes questions about different aspects of the program
An accurate statement about the post-test-only evaluation design is that it _____.
can be toned up with the addition of a comparison group
In the context of return on investment of a training program, the resource requirements model is used to _____.
compare costs of alternative training programs
The process of calculating the economic benefits of a training program with the help of accounting methods that consider costs and benefits involved in a training is known as ________ ___________ __________
cost-benefit analysis
The degree to which training outcomes are linked to the learned capabilities emphasized in a training program is known as _____.
criteria relevance
In an organizational context, the collection of information—including what, when, how, and from whom—for determining the effectiveness of a training program is referred to as the ___________.
evaluation design
Cognitive outcomes are related to _____ criteria in Kirkpatrick's framework.
level 2
The process of previewing a training program with expected trainees and managers or with other customers is known as __________.
pilot testing
Which of the following training evaluation designs is suitable for the companies where trainees are expected to have similar levels of knowledge, behavior, or results outcomes before training?
post-test only
Indra is the HR Manager at Plendon Architecture & Home Design Inc. Her company has been experiencing high levels of employee attrition over the last year, and most employees have attributed this to the low quality of training programs. To evaluate the training programs, Indra has decided to evaluate the knowledge levels of new employees by using a single test that will be conducted six months after their training. Indra has decided to use the ______ type of evaluation design.
post-test-only
The Solomon four-group evaluation design used for the evaluation of a company's training program is the combination of the pretest/post-test comparison group and the _____ design.
post-test-only control group
Companies that want to evaluate a training program without excluding certain employees or that want to train only a small group of employees often use the _____ design.
pretest/post-test
In the context of the evaluation process of a training program, planning and executing the evaluation involves _____.
previewing the program
In the context of methods used to minimize the threats to the validity of a company's training program evaluation, _________ _______________ refers to assigning employees to the training or comparison group on the basis of chance alone.
random assignment
Improvements in product quality or customer service are determined through _____ outcomes.
results
The process through which evaluation establishes to key business stakeholders that their hopes about training have been satisfied is known as _____.
return on expectations
To improve the strength of the time series training evaluation design, one can use _____________, a time period in which participants no longer receive training intervention
reversal
The training evaluation design determines _____.
the confidence that can be placed in the training results
Factors that require an evaluator to check the believability of the study results or the generalizability of the evaluation results to other groups of trainees and situations are known as ____________ _____________ _________
threats to validity
Jessica is a product development manager at Henry & Jameson Fashion Inc. A group of new employees recently completed six months at the company, including two months of training. To analyze if the training program they underwent is effective, Jessica has been conducting periodic evaluations of the employees' knowledge of the training material. She has been using the _____ method of evaluation design.
time series
In an organizational context, the process of gathering the outcomes required to determine the effectiveness of a training program is known as _______.
training evaluation
The measures that trainers and organizations use for the evaluation of training programs are known as ___________.
training outcomes