Chapter 6 Training Evaluation

One-time costs

Choice Costs related to needs assessment and program development

Which of the following training outcomes is used to ascertain the extent to which trainees know the facts, procedures, techniques, principles, and processes taught in the training program?

Cognitive outcomes

Identify the methods that are used to minimize the threats to the validity of a company's training program evaluation. (Check all that apply.)

Comparison groups pretest and post-tests random assignment

The Evaluation Process

Conduct a Needs Analysis Develop Measurable learning objectives and analyze transfer of training. Develop Outcome Measures Choose an evaluation strategy Plan and execute the evaluation

Costs per trainee

Costs related to meals, materials, and lost productivity incurred to replace trainees who are attending the training program

Learning or cognitive outcomes

Directly measure the extent to which trainees have mastered the training content

True or false: A reliable test allows the trainer to have confidence that any improvements in post-training test scores from pretraining levels are the result of test characteristics or the test environment.

False Reason: This is false. A reliable test allows the trainer to have confidence that any improvements in post-training test scores from pretraining levels are the result of learning that occurred in the training program, not test characteristics or the test environment.

True or false: Pilot testing is used for summative evaluation rather than formative evaluation.

False Reason: This is false. Pilot testing should be used for formative evaluation. It can be used as a dress rehearsal to show the program to managers, trainees, and customers.

True or false: Self-assessments rather than tests and quizzes are the preferred measures to assess cognitive outcomes.

False Reason: This is false. Tests and quizzes rather than self-assessments are the preferred measures of learning. This is because self-assessments are only moderately related to learning and influenced by how much learners liked the course or were motivated to learn rather than what they actually learned.

True or false: The use of workforce analytics is not necessary to use big data.

False Reason: This is false. Using big data requires the use of workforce analytics.

Identify an accurate statement about the resource requirements model.

It compares personnel, equipment, facilities, and materials costs across various stages of training.

Which of the following statements is true of the Solomon four-group evaluation design used for the evaluation of a company's training program?

It controls for most threats to internal and external validity.

Identify a characteristic of the pretest/post-test training evaluation design.

It fails to eliminate the effects of business conditions or other factors that affect the behavior of trainees.

Which of the following statements is true about the formative evaluation of a training program?

It gives information about the ways to make the training program better.

Identify an accurate statement about return on investment (ROI), which is measured for training and development programs.

It helps in ascertaining the economic value provided by these programs.

Internal validity

The believability of the study results.

Practicality

The convenience with which the training outcome measures can be accumulated

Criterion contamination

The degree that training outcomes are influenced by extraneous conditions or evaluate unsuitable capabilities

An evaluation of training that is conducted to ascertain the extent to which trainees have changed after participating in a training program is known as ______.

summative evaluation

The practice of using scientific and quantitative methods to examine data from various data sources to reach evidence-based decisions and show the effect of human resource practices on important company metrics is referred to as _____.

workforce analytics

Brian is a new employee at New Design Technologies, an interior design company. He underwent a three-month employee training and has been working for two months since the completion of the training. If his manager wants to analyze whether training Brian has generated economic value for the company, the manager should _____.

find out the profit gained by the company as a result of the clients attracted by Brian

Return on investment (ROI) analysis is suitable for employee training programs that are _____.

focused on an operational issue

The evaluation of training that takes place during the designing and development phase of a training program is called _____.

formative evaluation

In the context of Kirkpatrick's five-level framework for categorizing training outcomes, level 3 outcomes are collected to determine _____.

if the newly learned skills are being implemented on the job

In an organizational context, arrange the steps involved in the training evaluation process in the correct order of occurrence. (Place the first step at the top.)

1. Conducting a needs analysis 2. Developing measurable learning objectives and analyzing of training 3. Developing outcome measures 4. Selecting an eval strategy 5. Planning and executing evaluation

Arrange the steps involved in the process of determining the return on investment (ROI) of an employee training program in the correct order of occurrence. (Place the first step at the top.)

1. Understanding the objectives of the training program 2. Developing plans for collecting data related to the training objectives 3. Isolating the effects of the program from other factors that might affect the data related to the training objectives 4. Converting the data related to the training objectives to a monetary value and calculating ROI

What do skill-based training outcomes include? (Check all that apply.)

Acquisition of skills Application of skills on the job

___ _____ refers to complex data sets developed by compiling data across different organizational systems.

Big Data

Indirect Costs

General office supplies and administrative and staff support salaries.

The _____ refers to participants in an evaluation study performing at a high level simply because of the attention they are receiving and not due to the training.

Hawthorne Effect

Identify the true statements about random assignment, a method used to minimize the threats to the validity of a company's training program evaluation. (Check all that apply.)

It helps reduce the differences between the training group and comparison group in ability, knowledge, skill, or other personal characteristics. It helps ensure that the trainees are similar in individual characteristics such as age, gender, ability, and motivation.

Which of the following is true of the accounting method of determining costs involved in training programs?

It identifies when the costs are incurred.

Which of the following is true of summative evaluation?

It includes estimating an organization's monetary benefits from a training program.

Which of the following statements is true about the pretest/post-test with comparison group design for evaluating training?

It requires collection of pretraining and post-training outcome measures from trainee and comparison groups.

According to Kirkpatrick's five-level framework for categorizing training outcomes, which of the following levels of outcomes are collected at the completion of training, before trainees return to the job?

Level 2 outcomes Level 1 outcomes

Identify the levels of Kirkpatrick's framework that are related to affective training outcomes, depending on how they are evaluated.

Level 4 Level 3 Level 2

Pretraining measure

One way to improve the internal validity of the study results is is to first establish a baseline called...

Identify the training evaluation design that controls for most threats to validity.

Pretest/post-test with comparison group

Direct costs

Program material and supplies and salaries for all employees involved in training.

Reaction outcomes

Provide information regarding the extent to which the trainer, facilities, or learning environment may have hindered learning

Which of the following training evaluation outcomes is level 1 criteria in Kirkpatrick's five-level framework?

Reaction outcomes

Self assessments

Refers to a learners estimates of how much they know or have learned from training.

Summative evaluation

Refers to an evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program.

Return on investment

Refers to comparing the trainings monetary benefits with the cost of training.

Threats to validity

Refers to factors that will lead an evaluator to question either (1) the believability of the study results or (2) the extent to which the evaluation results are generalizable to other groups of trainees and situations

Training outcomes/criteria

Refers to measures that the trainer and the company use to evaluate training programs.

Training effectiveness

Refers to the benefits that the company and the trainees receive from training.

Evaluation design

Refers to the collection of information, including what, when, how, and from whom -- That will be used to determine the effectiveness of the training program.

Formative evaluation

Refers to the evaluation of training that takes place during program design and development.

Criteria relevance

Refers to the extent to which training outcomes are related to the learned capabilities emphasized in the training program.

Training evaluation

Refers to the process of collecting the outcomes needed to determine whether training is effective.

Pilot testing

Refers to the process of previewing the training program with potential trainees and managers or with other customers (persons who are paying for the development of the program).

Which of the following training outcomes involves comparing the monetary benefits of a training with its cost?

Return on investment

Which of the following training outcomes needs to be determined in order justify funding and to increase the status of a training and development function?

Return on investment

_____ refer to learners' estimates of how much they know or have learned from training.

Self-assessments

__________________ outcomes are used to assess the level of technical or motor skills and behaviors of trainees.

Skill-based

Identify the ways that determine the value of training using a design that can rule out and isolate its effects on results.

Success cases Return on expectations (ROE)

Discrimination

The extent to which the performance of trainees on the outcome reflects differences in performance on the job

Reliability

The extent to which training outcomes can be evaluated consistently over a time period

Criterion deficiency

The failure to evaluate training outcomes that were emphasized in the training objectives

Variety

The large number of data sources and different types of available data

Volume

The large quantity of available data

Velocity

The speed with which the large amount of generated data must be evaluated, captured, and made useful

Benefits

The value that the company gains from the training program.

Skill-based outcomes

These are used to assess the level of technical or motor skills and behaviors.

Cognitive outcomes

These are used to determine the degree to which trainees are familiar with the principles, facts, techniques, procedures, and processes emphasized in the training program.

Identify an accurate statement about affective training outcomes.

They involve trainees' motivation and attitudes.

Identify an accurate statement about reaction outcomes of a training program.

They refer to trainees' perceptions of the training program.

Affective outcomes

This definition includes attitudes and motivation.

Why is the training cost information important?

To compare the costs involved in other similar training programs To determine total expenses on training To determine the proportion of money spent on administration, training development, and evaluation

External Validity

Trainers interested in the generalizability of the study results to other groups and situations.

Costs Per Offering

Training site rental fees and trainer salaries

True or false: Reaction and cognitive outcomes of a company's training program do not help assess how much the trainees actually use the training content in their jobs.

True Reason: This is true. Reaction outcomes of a company's training program provide information regarding the extent to which the trainer, facilities, or learning environment may have hindered learning. Learning or cognitive outcomes directly measure the extent to which the trainees have mastered the training content. However, reaction and cognitive outcomes do not help determine how much the trainees actually use the training content in their jobs.

Identify the methods that tend to be helpful in identifying the benefits of employee training. (Check all that apply.)

Use of practitioner, technical, and academic literature Estimates of training benefits provided by trainees and their managers

Results

Used to determine the training programs payoff for the company.

Identify the cost-benefit analysis method that involves determining the dollar value of training on the basis of different factors such as estimates of the difference in job performance between trained and untrained employees and the number of individuals trained.

Utility analysis

Utility analysis uses _____ to estimate the difference in job performance for trained versus untrained employees.

a pretest/post-test design with a comparison group

Results outcomes are used to ascertain _____.

a training program's payoff for an organization

Hundred new employees of HFCT Bank completed the bank's employee training program. Harold, the HR manager, wants to know if the employees liked the training program. In this scenario, Harold should _____.

ask them to take a survey that includes questions about different aspects of the program

An accurate statement about the post-test-only evaluation design is that it _____.

can be toned up with the addition of a comparison group

In the context of return on investment of a training program, the resource requirements model is used to _____.

compare costs of alternative training programs

The process of calculating the economic benefits of a training program with the help of accounting methods that consider costs and benefits involved in a training is known as ________ ___________ __________

cost-benefit analysis

The degree to which training outcomes are linked to the learned capabilities emphasized in a training program is known as _____.

criteria relevance

In an organizational context, the collection of information—including what, when, how, and from whom—for determining the effectiveness of a training program is referred to as the ___________.

evaluation design

Cognitive outcomes are related to _____ criteria in Kirkpatrick's framework.

level 2

The process of previewing a training program with expected trainees and managers or with other customers is known as __________.

pilot testing

Which of the following training evaluation designs is suitable for the companies where trainees are expected to have similar levels of knowledge, behavior, or results outcomes before training?

post-test only

Indra is the HR Manager at Plendon Architecture & Home Design Inc. Her company has been experiencing high levels of employee attrition over the last year, and most employees have attributed this to the low quality of training programs. To evaluate the training programs, Indra has decided to evaluate the knowledge levels of new employees by using a single test that will be conducted six months after their training. Indra has decided to use the ______ type of evaluation design.

post-test-only

The Solomon four-group evaluation design used for the evaluation of a company's training program is the combination of the pretest/post-test comparison group and the _____ design.

post-test-only control group

Companies that want to evaluate a training program without excluding certain employees or that want to train only a small group of employees often use the _____ design.

pretest/post-test

In the context of the evaluation process of a training program, planning and executing the evaluation involves _____.

previewing the program

In the context of methods used to minimize the threats to the validity of a company's training program evaluation, _________ _______________ refers to assigning employees to the training or comparison group on the basis of chance alone.

random assignment

Improvements in product quality or customer service are determined through _____ outcomes.

results

The process through which evaluation establishes to key business stakeholders that their hopes about training have been satisfied is known as _____.

return on expectations

To improve the strength of the time series training evaluation design, one can use _____________, a time period in which participants no longer receive training intervention

reversal

The training evaluation design determines _____.

the confidence that can be placed in the training results

Factors that require an evaluator to check the believability of the study results or the generalizability of the evaluation results to other groups of trainees and situations are known as ____________ _____________ _________

threats to validity

Jessica is a product development manager at Henry & Jameson Fashion Inc. A group of new employees recently completed six months at the company, including two months of training. To analyze if the training program they underwent is effective, Jessica has been conducting periodic evaluations of the employees' knowledge of the training material. She has been using the _____ method of evaluation design.

time series

In an organizational context, the process of gathering the outcomes required to determine the effectiveness of a training program is known as _______.

training evaluation

The measures that trainers and organizations use for the evaluation of training programs are known as ___________.

training outcomes