Psyc 517: Chapter 6 - Training Evaluation

Benefits

are the value that the company gains from the training program

Pre-tests

one way to improve the internal validity of the study results is to first establish a baseline (or pre-training measure) of the outcome

Types of Evaluation Designs

1) Post-test only 2) Pretest/post-test 3) Pretest/Post-test with comparison group 4) Time series 5) Solomon four-group

Methods to Control for Threats to Validity

1) Pre-tests and Post-tests 2) Use of Comparison Groups 3) Random Assignment

Why should a training program be evaluated?

1) To identify the program's strengths and weaknesses. This includes determining if the program is meeting the learning objectives, if the quality of the learning environment is satisfactory, and if transfer of training to the job is occurring 2) To assess whether the content, organization, and administration of the program— including the schedule, accommodations, trainers, and materials—contribute to learning and the use of training content on the job 3) To identify which trainees benefit from the program 4) To assist in marketing programs through the collection of information from participants about whether they would recommend the program to others, why they attended the program, and their level of satisfaction with the program 5) To determine the financial benefits and costs of the program 6) To compare the costs and benefits of training versus non-training investments (such as work redesign or a better employee selection system) 7) To compare the costs and benefits of different training programs to choose the best program

The Evaluation Process

1. Determine training needs by conducting needs analysis 2. Identify specific, measurable training objectives to guide the program 3. Develop outcome measures 4. Choose an evaluation strategy 5. Plan and execute the evaluation

Six categories of training outcomes:

1. Reaction outcomes 2. Learning or Cognitive outcomes 3. Behavior and Skill-based outcomes 4. Affective outcomes 5. Results outcomes 6. Return on Investment (ROI) outcomes

Considerations in Choosing an Evaluation Design

1. The evaluation results can be used to change the program 2. The training program is ongoing and has the potential to have an important influence on employees or customers 3. The training program involves multiple classes and a large number of trainees 4. Cost justification for training is based on numerical indicators (here the company has a strong orientation toward evaluation) 5. Trainers or others in the company have the expertise (or the budget to purchase expertise from outside the company) to design and evaluate the data collected from an evaluation study

Training cost information is important for several reasons:

1. To understand total expenditures for training, including direct and indirect costs 2. To compare the costs of alternative training programs 3. To evaluate the proportion of money spent on training development, administration, and evaluation as well as to compare monies spent on training for different groups of employees (exempt versus nonexempt, for example) 4. To control costs

Kirkpatrick's five-level framework includes:

Level 1: Reaction outcomes Level 2: Learning or cognitive outcomes Level 3: Behavior/Skill outcomes Level 4: Results Level 5: Return on Investment (ROI)

Pilot testing

Process of previewing the training program with potential trainees and managers or with other customers

Summative evaluation

an evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program — Used to determine whether trainees have acquired knowledge, skills, attitudes, behavior, or other outcomes identified in the training objectives — May also include measuring the monetary benefits (ROI) that the company receives from the program — Involves collecting quantitative (numerical) data through tests, ratings of behavior, or objective measures of performance such as volume of sales, accidents, or patents

Post-tests

another measure of the outcomes that can be taken after training (or post-training measure)

Behavior/skill outcomes

are collected to determine whether behavior/skills are being used on the job

Outcomes

are not perfectly related to each other

Indirect costs

are not related directly to the design, development, or delivery of the training program; they include general office supplies, facilities, equipment, and related expenses; travel and expenses not directly billed to one program; training department management and staff salaries not related to any one program; and administrative and staff support salaries

Skill-based outcomes

are used to assess the level of technical or motor skills and behaviors; they include acquisition or learning of skills (skill learning) and use of skills on the job (skill transfer)

Learning or cognitive outcomes

are used to determine the degree to which trainees are familiar with the principles, facts, techniques, procedures, and processes emphasized in the training program; measure what knowledge trainees learned in the program

Results

are used to determine the training program's payoff for the company

Both level 1 and level 2 outcomes (reactions and learning) are collected

at the completion of training, before trainees return to the job

Training effectiveness

benefits that the company and the trainees receive from training; benefits for trainees may include learning new skills or behaviors

Training evaluation involves

both formative and summative evaluation

Solomon four-group

combines the pre-test/post-test comparison group and the post-test-only control group design; in this design, a training group and a comparison group are measured on the outcomes both before and after training

To ensure adequate training evaluation

companies should collect outcome measures related to both learning and transfer

To evaluate its training program, a company must decide

how it will determine the program's effectiveness; that is, it must identify what training outcomes or criteria it will measure

How long after training should outcomes be collected?

in most cases, reactions are measured immediately after; learning, behavior, and results should be measured after sufficient time has elapsed to determine whether training has had an influence on these outcomes

Affective outcomes

include attitudes and motivation; outcomes that might be collected in an evaluation include tolerance for diversity, motivation to learn, safety attitudes, and customer service orientation; these outcomes can be measured using surveys

Direct costs

include salaries and benefits for all employees involved in training, including trainees, instructors, consultants, and employees who design the program; program material and supplies; equipment or classroom rentals or purchases; and travel costs

Utility analysis

is a cost-benefit analysis method that involves assessing the dollar value of training based on estimates of the difference in job performance between: • trained and untrained employees • the number of individuals trained • the length of time a training program is expected to influence performance, and • the variability in job performance in the untrained group of employees

Cost-benefit analysis

is the process of determining the economic benefits of a training program using accounting; the benefits of the training can be identified by considering the objectives of the training program and the type of outcomes the program was to influence

What the evaluator strives for

is to use the most rigorous design possible (given the circumstances under which the evaluation occurs) to rule out alternative explanations for the results of the evaluation

Kirkpatrick's five-level framework

is used for categorizing training outcomes, and a description of each of the outcomes and how they are measured

Training outcomes or criteria

measures that the trainer and the company use to evaluate training programs

Success cases

refer to concrete examples of the impact of training that show how learning has led to results that the company finds worthwhile and the managers find credible; they do not attempt to isolate the influence of training but rather to provide evidence that it was useful

Threats to validity

refer to factors that will lead an evaluator to question either (1) the believability of the study results or (2) the extent to which the evaluation results are generalizable to other groups of trainees and situations

Reaction outcomes

refer to trainees' perceptions of the program, including the facilities, trainers, and content; often referred to as a "creature comfort"; is collected at the program's conclusion

Dashboard

refers to a computer interface designed to receive and analyze the data from departments within the company to provide information to managers and other decision-makers

Comparison Group

refers to a group of employees who participate in the evaluation study but do not attend the training program; the comparison employees have personal characteristics (e.g., gender, education, age, tenure, skill level) as similar to the trainees as possible; internal validity can be improved by using a control (or comparison group)

Reversal

refers to a time period in which participants no longer receive the training intervention

Pretest/Post-test

refers to an evaluation design in which both pre-training and post-training outcome measures are collected; there is no comparison group, which makes it difficult to rule out the effects of business conditions or other factors as explanations for changes

Post-test Only

refers to an evaluation design in which only post-training outcomes are collected; this design is appropriate when trainees (and the comparison group, if one is used) can be expected to have similar levels of knowledge, behavior, or results outcomes (e.g., same number of sales or equal awareness of how to close a sale) prior to training

Time series

refers to an evaluation design in which training outcomes are collected at periodic intervals both before and after training; one advantage of this design is that it allows an analysis of the stability of training outcomes over time; the strength of this design can be improved by using reversal

Pretest/Post-test with Comparison Group

refers to an evaluation design that includes trainees and a comparison group; if improvement is greater for the training group than the comparison group, this finding provides evidence that training is responsible for the change; this type of design controls for most threats to validity

Random Assignment

refers to assigning employees to the training or comparison group on the basis of chance alone; employees are assigned to the training program without consideration of individual differences (ability or motivation) or prior experiences; this helps to ensure that trainees are similar in individual differences such as age, gender, ability, and motivation

Return on Investment (ROI)

refers to comparing the training's monetary benefits with the cost of the training

Hawthorne effect

refers to employees in an evaluation study performing at a high level simply because of the attention they are receiving; use of a comparison group helps to show that any effects observed are due specifically to the training rather than the attention the trainees are receiving

Workforce analytics

refers to the practice of using quantitative methods and scientific methods to show that HR practices (including training and development) influence important company metrics (outcomes)

Return on Expectations (ROE)

refers to the process through which evaluation demonstrates to key business stakeholders, such as top-level managers, that their expectations about training have been satisfied

Internal threats to validity

relate to characteristics of the company (history), the outcome measures (instrumentation, testing), and the persons in the evaluation study (maturation, regression toward the mean, mortality, initial group differences); these characteristics can cause the evaluator to reach the wrong conclusions about training effectiveness

External threats to validity

relate to how study participants react to being included in the study and the effects of multiple types of training

Affective outcomes

relate to trainees' attitudes toward training content and motivation to transfer

Reactions

relates to trainees providing feedback about their satisfaction with a trainer

Internal validity

the believability of the study results

Evaluation Design

the collection of information—including what, when, how, and from whom—that will be used to determine the effectiveness of the training program

The design of the training evaluation determines

the confidence that can be placed in the results; that is, how sure a company can be that training is either responsible for changes in evaluation outcomes or has failed to influence the outcomes; no evaluation design can ensure that the results of the evaluation are completely due to training

Reliability

the degree to which outcomes can be measured consistently over time — A reliable test includes items for which the meaning or interpretation does not change over time — A reliable test allows the trainer to have confidence that any improvements in post-training test scores from pre-training levels are the result of learning that occurred in the training program, not test characteristics

Discrimination

the degree to which trainees' performance on the outcome actually reflects true differences in performance; the test should discriminate on the basis of trainees' knowledge of electrical principles

Practicality

the ease with which the outcome measures can be collected

Formative evaluation

the evaluation of training that takes place during program design and development — Ensures that the training program is well organized and runs smoothly and that trainees learn and are satisfied with the program — Provides information about how to make the program better — Asks customers, employees, managers, and subject-matter experts (SMEs) their opinions on the description of the training content and objectives and the program design — Is conducted either individually or in groups before the program is made available to the rest of the company — Pilot testing is used

Criterion contamination

the extent that training outcomes measure inappropriate capabilities or are affected by extraneous conditions — Criteria may be contaminated if the conditions under which the outcomes are measured vary from the learning environment

External validity

the extent to which the evaluation are generalizable to other groups of trainees and situations

Criteria relevance

the extent to which training outcomes are related to the learned capabilities emphasized in the training program — The learned capabilities required to succeed in the training program should be the same as those required to be successful on the job — Outcomes need to be valid measures of learning

Criterion deficiency

the failure to measure training outcomes that were emphasized in the training objectives

Training evaluation

the process of collecting the outcomes needed to determine whether training is effective

Which training outcomes measure is best?

this depends on the training objectives

Level 3, 4, and 5 outcomes can be used to determine

whether trainees are using training content back on the job (i.e., whether transfer of training has occurred)

Level 4 and 5 outcomes (results and return on investment) can be used to determine

whether training has resulted in an improvement in business results, such as productivity or customer satisfaction; these criteria also help to determine whether the benefits of training exceed their costs

Appropriate training outcomes need to be:

— Relevant — Reliable — Discriminative — Practical

Reasons for Evaluating Training

• Training evaluation provides a way to understand the investments that training produces and provides information needed to improve training • Training evaluation provides the data needed to demonstrate that training does offer benefits to the company