Training Evaluation

Evaluating Mistakes to Avoid

-addressing evaluation requirements after the program has launched -spending the majority of evaluation resources on Levels 1 and 2 -relying solely on standardized surveys -not using collected data

Level 2

-in Harry Potter, the OWLs examine Level 2 training outcomes -assessment of the skills learned during training is level 2 -select the best answer to each question below.... A. B. C. D. PROS and CONS of relying on self-assessments to evaluate cognitive outcomes? -self-assessments are only moderately related to learning -self-assessments tend to be biased by how much learners liked the course or were motivated to learn -tests and quizzes are the preferred method, though self-assessments are more common

Kirkpatrick's Model of Evaluation

-prevailing categorization of outcomes that can be used to evaluate training -Ranges from level 1 (reactions) to level 5 (ROI) Kirkpatrick and Kirkpatrick (2015): when creating a post-training evaluation plan, begin with the end in mind. Outcomes for training evaluation- how should a company determine the effectiveness of a program? What outcomes will it measure?

Level 4- Results

-results outcomes are to determine the training program's payoff for the company ex. increased production and reduced costs related to top talent turnover rates, accidents, equipment downtime, improvements in product quality or customer service -an analysis of top talent turnover rates is a level 4 outcome

Level 2-Cognitive

-the knowledge that trainees learned in the program (familiarity with the principles, facts, procedures emphasized in training) -typically collected at the end of training, before trainees return to the job -can be measured with tests, quizzes, or self-assessments Level 2 outcomes

Level 3- Behavior

-the level of technical or motor skills and behaviors acquired through the training program includes both skill learning and skill transfer -can be evaluated by observing performance in training (work sample, simulation) or on-the-job classified as Level 2 (learning) and Level 3 (behavior) outcomes Kirkpatrick and Kirkpatrick (2016) Level 3 is the most important level to evaluate and invest in. -who provides the most accurate assessment of skill or behavior transfer of training? -trainees, subordinates, managers, peers! -self-rating alone create inaccurate positive assessments -the best assessment comes from a combination of sources -please check all responses that apply to you. -consider your opportunities over the past 3 months to observe and interact with the principal you are rating. Read the definition and behaviors associated with the skill. Then complete your rating using the following scale: 1. never 2. seldom 3. sometimes 4. usually 5. always

training outcomes or criteria

-the specific measures that the trainer and company use to evaluate training programs

Training Evaluation

the process of collecting outcomes needed to determine whether training is effective -training outcomes or criteria -evaluation design

skill transfer

the use of skills on the job

Selecting a Training Outcome

although it may be ideal to collect all five levels out outcomes, it is not always necessary. Guidelines: -let training program objectives guide this decision -criteria should be appropriate (relevant, reliable and practical) -collect outcome measures for both learning (Levels 1 and 2) and transfer of training (Levels 3,4, and 5)

Level 1 evaluation

can help identify problems with the training process and address them -is particularly relevant for affective learning outcomes and with attitude training (such as diversity, ethics, motivation) -yet, happiness does not necessarily mean that training is a success

summative evaluation

evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program -typically involves an evaluation of whether trainees have acquired outcomes identified in the training objectives (knowledge, skills, attitudes, behaviors) -may also include measuring the monetary benefits (ROI) the company receives -usually involves quantitative data (tests, ratings, or objectives measures of performance)

Relevance

extent to which training outcomes are related to the learning capabilities emphasized in the training program -also referred to as validity involves minimizing two concerns -contamination -deficiency -failture to measure training outcomes that were emphasized in the training objectives

deficiency

failure to measure training outcomes that were emphasized in the training objectives

Affective outcomes

include attitudes and motivation -can be measured using surveys -classified as Level 2 (learning), Level 3 (behavior), and Level 4 (results) outcomes (depending on evaluation Ex. training on tolerance for diversity, employee engagement, motivation to learn, safety attitudes, customer service orientation -training focused on employee career goals, plans, and interests 1. strongly disagree 2. disagree 3. neutral 4. agree 5. strongly agree

self-assessments

learners' estimates of how much they know or have learned from training

contamination

reel extent that training outcomes measure inappropriate capabilities or are affected by extraneous conditions

skill learning

the acquisition or learning of skills

training effectiveness

the benefits that the company and trainees receive from training For example: -Trainees may benefit in terms of learning new skills and behaviors -Companies may benefit in terms of increased sales or more satisfied customers

evaluation design

the collection of information (including what, when, how, and from whom) that will be used to determine the effectiveness of the training program

Reliability

the degree to which outcomes can be measured consistently over time. -the meaning or interpretation of items does not change over time -provides trainer confidence that any improvements in outcomes are the result of learning and not test characteristics or the test environment

Practicality

the ease with which the outcomes measures of training effectiveness can be collected

Learning (Levels 1 and 2)

Reactions- did trainees like the program? Did the environment help learning? Was the material meaningful? Cognitive- pencil-and-paper tests Behavior- performance on a work sample

Level 5- ROI

Return on investment -comparing the training's monetary benefits with the cost of training -originally Kilpatrick's model of evaluation had 4 levels, but it has been extended to list ROI as a distinct outcome type (Level 5)

Level 1-reactions

Trainees' perceptions of the program (including facilities, trainers, and content) -also referred to as class or instructor evaluations, evaluations of creature comforts, or a smile sheet -typically collected with questionnaires at the end of the training program, before trainees return to the job -should include all factors related to a successful learning environment Level 1 outcome -Lopker and Askeland (2009) Level 1 evaluation offers more relevant information than you might expect.

formative evaluation

Evaluation of training that takes place during program design and development -helps ensure that the program is organized and runs smoothly and trainees learn and are satisfied with the program -provides information on how to make the program better -usually involve qualitative data (opinions, beliefs, and feelings about the program) -data is collected individually or in groups before the program is launched -may involve pilot testing: previewing with small group of employees

Before and After

Formative is before -qualitative Summative is after -quantitative

Outcomes for Training Evaluation

Percentage of Companies using outcomes Reaction- 92% Cognitive- 81% Behavior- 55% Results- 37% ROI- 18% None- 4%

validity

also referred to as relevance

Transfer of Training

Behavior-Ratings by peers or managers based on observation of behavior Affective- trainees' motivation, job attitudes, or engagement Results- did company benefit through sales, quality, productivity, reduced accidents and complaints? -performance on work equipment