Chapter 6
pretraining measure
A baseline measure of outcomes.
pre-tests and post-tests
used to determine the extent to which trainees have changed from pre-training to post-training measures.
return on investment (ROI)
A comparison of a training program's monetary benefits and costs.
benefits
The value a company gains from a training program.
"pencil-and-paper" tests
Typically, ________ are used to assess cognitive outcomes.
control group
Used to rule out factors other than training as the cause of changes in the trainees.
post-test only
An evaluation design in which only post-training outcomes are collected.
Net Promoter Score (NPS)
A score designed to measure satisfaction with a training course by asking trainees to rate the likelihood of recommending it to a peer by using a 0-10 point scale.
reversal
A time period in which training participants no longer receive training intervention.
practicality
The ease with which outcome measures can be collected.
volume
refers to the amount of available data.
velocity
refers to the huge amount of data that is being generated and the speed with which it must be evaluated, captured, and made useful.
affective learning outcomes
relate to attitudes toward specific training content areas, such as diversity and teamwork, and individuals' motivation to transfer skills upon completion of training.
reactions
relate to the trainees' perceptions of the content, facilities, trainer, and methods of delivery of the training experience.
reactions
relates to trainees' perceptions of the training experience, and affective learning outcomes relate to attitudes and motivation that are a focus of training.
indirect costs
Costs not related specifically to a training program's design, development, or delivery.
observations
Behavior and skill-based outcomes are best measured by ________.
external validity
If trainers are interested in the generalizability of a study's results, they are said to be interested in the ________ of the study.
velocity
In the context of big data, ________ refers to the huge amount of data that is being generated and the speed with which it must be evaluated, captured, and made useful.
ROI
It is estimated that 15% of organizations collect data on which of the following training outcomes?
results
It is estimated that 35% of organizations collect data on which of the following training outcomes?
self-evaluation
Learners' estimates of how much they know or have learned from training.
training outcomes
Measures that a company and its trainer use to evaluate training programs. Also called criteria.
criteria
Measures that a company and its trainer use to evaluate training programs. Also called training outcomes.
affective outcomes
Outcomes including attitudes and motivation.
cognitive outcomes
Outcomes that are used to measure what knowledge trainees learned in a training program.
skill-based outcomes
Outcomes used to assess the level of technical or motor skills or behavior; outcomes include skill acquisition or learning and on-the-job use of skills.
results
Outcomes used to determine a training program's payoff.
they represent a group of employees who do not attend training.
Which of the following statements is most true of comparison groups?
time series
A company that aims to improve readily observable outcomes by collecting data at periodic intervals before and after training is likely applying a ________ evaluation design.
dashboard
A computer interface designed to receive and analyze the data from departments within the company to provide information to managers and other decision makers.
significant financial investments
Return on investment (ROI) analysis is best suited for training programs that are ________.
pre-training
The ______ measure essentially establishes a baseline.
random assignment
The assignment of employees to training or a comparison group on the basis of chance.
training effectiveness
The benefits that a company and its trainees receive from training.
discrimination
The degree to which trainees' performances on an outcome actually reflect true differences in performance.
conducting a needs analysis
The evaluation process ideally begins with ________.
criteria relevance
The extent to which training outcomes relate to the learned capabilities emphasized in training.
criterion deficiency
The failure to measure training outcomes that were emphasized in training objectives.
external validity
The generalizability of study results to other groups and situations.
workforce analytics
The practice of using quantitative methods and scientific methods to analyze data from human resource databases, corporate financial statements, employee surveys, and other data sources to make evidence-based decisions and show that human resource practices (including training, development, and learning) influence important company metrics.
training evaluation
The process of collecting the outcomes needed to determine whether training has been effective.
cost-benefit analysis
The process of determining the economic advantages of a training program using accounting methods.
pilot testing
The process of previewing a training program with potential trainees and managers or other customers.
return on expectations (ROE)
The process through which evaluation demonstrates to key business stakeholders, such as top-level managers, that their expectations about training have been satisfied.
logic model
The process used to identify the relationship between training resources, training activities, and program outcomes.
volume, variety, and velocity
What are the characteristics of big data?
To identify the program's strengths and weaknesses, including whether the program is meeting the learning objectives, the quality of the learning environment, and if transfer is occurring; To assess whether the various features of the training context and content contribute to learning and transfer; To identify which trainees benefited most or least from the program and why; To gather information, such as testimonials, to use for marketing training; To determine financial benefits and costs of the program; To compare the costs and benefits of training versus other HRM investments; To compare the costs and benefits of various training programs in order to choose the most effective programs
What are the various reasons for evaluating the effectiveness of training programs?
the use of pre-tests and post-tests in evaluation designs; control groups; and random assignment.
What are the ways to minimize threats to validity?
criterion contamination
When a training program's outcomes measure inappropriate capabilities or are affected by extraneous conditions.
greater appreciation of diversity
Which of the following is an affective learning outcome?
formative evaluation focuses on how to make a training program better, whereas summative evaluation helps to determine the extent to which trainees have changed after training.
Which of the following statements best differentiates formative evaluation from summative evaluation?
trainees are categorized into detractors, permissives, and promotors.
Which of the following statements is NOT true regarding a Net Promoter Score (NPS)?
it combines both pre-test/post-test comparison group and post-test-only/control group designs.
Which of the following statements is true of Solomon four-group evaluation design?
it assigns employees to a training program without considering their individual differences.
Which of the following statements is true of random assignment?
reactions
Which training outcome relates to trainees providing feedback about their satisfaction with a trainer?
reliability
Written materials' level of difficulty.
return on expectations (ROE)
________ demonstrate(s) to key business stakeholders, such as top-level managers, that their expectations about training have been satisfied.
reactions
________ is (are) likely to be the easiest training outcome to measure.
reversal
________ is a time period in which participants no longer receive training intervention.
success cases
________ refers to concrete examples to show how learning has led to results that a company finds worthwhile and credible.
practicality
________ refers to the ease with which training outcomes can be collected.
training evaluation
________ refers to the process of collecting the outcomes needed to determine whether training is effective.
discrimination
________ refers to whether performance on the outcome reflects true differences in performance.
affective outcomes
________ relate to trainees' attitudes toward training content and motivation to transfer.
behavior and skill-based outcomes
________ relate(s) to whether trainees are using training content back on the job.
Random assignment of employees
helps to ensure that members of the control group and training group are of similar makeup prior to the training.
variety
includes the large number of sources and types of data.
post-training measure
A measure of outcomes taken after training.
variety
In the context of big data, ________ refers to the large number of sources and types of data.
utility analysis
A cost-benefit analysis method that involves assessing the dollar value of training based on estimates of the difference in job performance between trained and untrained employees, the number of individuals trained, the length of time a training program is expected to influence performance, and the variability in job performance in the untrained group of employees.
comparison group
A group of employees who participate in an evaluation study but do not attend a training program.
Hawthorne effect
A situation in which employees in an evaluation study perform at a high level simply because of the attention they are receiving.
reaction outcomes
A trainee's perceptions of a training program, including perceptions of the facilities, trainers, and content.
self-assessments
An employee's use of information to determine career interests, values, aptitudes, and behavioral tendencies.
Solomon four-group
An evaluation design combining the pretest/post-test comparison group and the post-test-only control group designs.
pretest/post-test
An evaluation design in which both pretraining and post-training outcome measures are collected.
time series
An evaluation design in which training outcomes are collected at periodic intervals pre- and post-training.
pretest/post-test with comparison group
An evaluation design that includes trainees and a comparison group. Both pretraining and post-training outcome measures are collected.
trainees' performance on the outcome should actually reflect true differences in performance
Appropriate training outcomes need to discriminate, meaning that ________.
big data
Complex data sets characterized by volume, variety, and velocity that are developed by compiling data across different organizational systems, including marketing and sales, human resources, finance, accounting, customer service, and operations.
success cases or stories
Concrete examples of the impact of training that show how learning leads to results that the company finds worthwhile and the managers find credible.
direct costs
Costs that are actually connected to training, including the salaries and benefits of all employees involved, program supplies, equipment and classroom rental or purchase, and travel costs.
evaluation design
Designation of what information is to be collected, from whom, when, and how to determine the effectiveness of training.
internal validity
Establishing that a treatment (training) made a difference.
formative evaluation
Evaluation conducted to improve the training process; usually conducted during program design and development.
a company is only interested in whether trainees have achieved a certain proficiency level
Evaluation designs without pre-test or comparison groups are most appropriate when ________.
summative evaluation
Evaluation of the extent that trainees have changed as a result of participating in a training program.
threats to validity
Factors that lead one to question either (1) the believability of a study's results or (2) the extent to which evaluation results are generalizable to other groups of trainees and situations.
their opinions and feelings about the program
Formative evaluation involves collecting data about a training program from trainees mainly through ________.
behavior and skill-based outcomes
Front-line supervisors are likely most concerned with which training outcomes?
Data can be used from different sources to demonstrate relationships between learning and business outcomes, such as data from employee records, learning and performance management systems, and customer relationship management systems. The goal is to merge the data from these sources to show that employees who attended training had positive influence of different metrics.
How can big data be used to show that learning influences business outcomes?
The following should be considered: The evaluation results can be used to change the program; The training is ongoing and has the potential to affect many employees.; The training program involves multiple classes and a large number of trainees.; Cost justification for training is based on numerical indicators; Trainers or others in the company have the expertise to evaluate.; The cost of training creates a need to show that it works.; There is sufficient time for conducting an evaluation.; There is interest in measuring change from pre-training levels or in comparing two or more
Identify the circumstances when a company should consider employing a more rigorous design for evaluating a training program.
results
If a firm measures its sales volume before and after a training program, which training outcome is it focused on?