Training and Development Ch. 6

Evaluation Design

Collection of information, including whom, what, when, and how, for determining the effectiveness of the training program

Direct Costs

include salaries and benefits for all employees incvolved in training including trainees, instructors, consultants, and employees who design the program; program material and supplies; equipment or classroom rentals or purchases and travel costs

TQI

is a computer application that collects data about training department performance, productivity, budget, and courses and allows for detailed analysis of the data. TQI tracks all determents training data into five categories: effectiveness, quantity, perceptions, financial impact, and operational impact

Skill-based outcomes

used to asses the level of technical or motor skill and behaviors includes acwusion or learning skills and use of skills on the job. skill based outcomes relate to Kirkpatricks level 2 and level 3

cognitive outcomes

used to determine the degree to which trainees are familiar with principles, facts, technologies, procedures, or processes emphasized in the training program. measure what knowledge trainees learned in the program

Positive transfer of training is demonstrated

when learning occurs and positive changes in skill-based, affective, or results outcomes are also observed

reversal

which refers to a time period in which participants no longer receive the training intervention.

One way to improve the internal validity of the study results is to

first establish a baseline or pretraining measure of the outcome

No transfer of training is demonstrated

if learning occurs but no changes are observed in skill- based, affective, or learning outcomes.

Cost-benefit analysis

in this situation is the process of determining the economic benefits of a training program using accounting methods that look at training costs and benefits

affective outcomes

include attitudes and motivation. affective outcomes that might be collected in an evaluation include tolerance for diversity, motivation to learn, safety attitudes, and customer service orientation. measured using surveys

Measuring Human Capitol and Training Activity

- American society of training and development (ASTD): provides information about training hours and delivery methods that companies can use to benchmark - Workforce Analytics: practice of using quantitative methods and scientific methods to analyze data from human resource databases and other databases to influence important company metrics - dashboards: computer interface designed to receive and analyze the data from departments within the company to provide information to managers and other decision makers 1. useful because they can provide a visual display using charts of the relationship between learning activities and business performance data.

Reasons for Evaluating Training

- Companies make large investments in training and education and view them as a strategy to be successful; they expect the outcomes of training to be measurable - Training evaluation provides the data needed to demonstrate that training does provide benefits to the company 1. It involves formative and summative evaluation

Determining Return on Investment

- Cost- Benefit analysis: Process of determine the economic benefits of training program using accounting methods that look at training costs and benefits - ROI should be limited only to certain training programs, because it can be costly - determine costs 1. methods for comparing costs of alternative training programs include the resource requirements model and accounting - determine benefits - Methods include: 1. technical, academic, and practitioner literature 2. Pilot training programs and observance of successful job performers 3. observance of successful job performers 4. estimates by trainees and their managers - to calculate ROI 1. identify outcomes 2. place a value on the outcomes 3. determine the change in performance after eliminating other potential influences on training results 4. Obtain an annual amount of benefits 5. determine the training costs 6. calculate the total benefits by subtracting the training costs from benefits 7. calculate the ROI by dividing operational results by costs a. The ROI gives an estimate of dollar return expected from each dollar invested in training

summative evaluation

- determines the extent to which trainees have changed as a result of participating in the training program 1. it may include measuring the monetary benefits that the company received from the program (ROI) 2. it involves collecting quantitative data - a training program should be evaluated: 1. to identify the programs strengths and weaknesses 2. to assess whether content organization, and administration of the program contribute to learning and the use of training content on the job 3. to identify which trainees benefited most or least from the program 4. to gather data to assist in marketing training programs 5. to determine the financial benefits and costs of the program 6. to compare the costs and benefits of: a. training versus non-training investments b. different training programs to choose the best programs

evaluation practices

- it is important to recognize limitations of choosing to measure only reaction and cognitive outcomes 1. to ensure an adequate training evaluation, companies must collect outcome measures related to both learning and transfer

types of evaluation designs

- post- test only: only post-training outcomes are collected 1. appropriate when trainees can be expected to have similar levels of knowledge, behavior, or results outcomes prior to training - pretest / post-test: pre-training and post-training outcomes measures are collected 1. used by companies that want to Eva;uate a training program but are comfortable with excluding certain employees - pretest / post-test with comparison group: includes trainees and a comparison group. 1. differences between each of the training conditions and the comparison group are analyzed determining whether differences between the groups were caused by training - time series: training outcomes are collected at periods intervals both before and after training 1. it allows an analysis of the stability of training outcomes over time 2. reversal: time period in which participants no longer receive the training intervention - Solomon four group: combines the pretest / post-test comparison group and the post-test only control group design 1. this design controls for most threats to internal and external validity

Methods to control for threats to validity

- protests and post-tests: comparison of the post-training and pertaining measures can indicate the degree to which trainees have changed as a result of training - use of comparison groups: groups of employees who participate in the evaluation study but do not attend the training program 1. Hawthorne effect - Random assignment: assigning employees to the training or comparison group on the basis of chance alone 1. it is often impractical 2. analysis of covariance

formative evaluation

- takes place during program design and development 1. it helps ensure that the training program is well organizes runs smoothly 2. trainees learn and are satisfied with the program - it provides information about how to make the program better; it involves collecting qualitative data about the program Pilot testing: process of previewing the training program with potential trainees and managers or with other customers

evaluation designs

- threats to validity: factors that will lead an evaluator to question ether the: 1. internal validity: the reliability o fetch study results 2. external validity: the extent to which the evaluation results are generalizable to the regroups of trainees and situations

outcomes used in the evaluation of training programs (cont 2)

- training quality index (TQI): computer application that collects data about training departments performance, productivity, budget, and courses, and allow detailed analysis of this data 1. quality of training is included in the effeteness category

From the discussion of summative and formative evaluation, it is probably apparent to you why a training program should be evaluated:

1. To identify the program's strengths and weaknesses. This includes determining if the program is meeting the learning objectives, if the quality of the learning environment is satisfactory, and if transfer of training to the job is occurring. 2. To assess whether the content, organization, and administration of the program— including the schedule, accommodations, trainers, and materials—contribute to learn- ing and the use of training content on the job. 3. To identify which trainees benefit most or least from the program. 4. To assist in marketing programs through the collection of information from participants about whether they would recommend the program to others, why they attended the pro- gram, and their level of satisfaction with the program. 5. To determine the financial benefits and costs of the program. 6. To compare the costs and benefits of training versus nontraining investments (such as work redesign or a better employee selection system). 7. To compare the

the evaluation process

1. conduct a needs analysis 2. develop measurable learning objectives and analyze transfer of training 3. develop outcome measures 4. choose an evaluation strategy 5. plan and execute the evaluation

Another measure of the outcomes can be taken after training. This is referred to as a posttraining measure.

A comparison of the posttraining and pretraining measures can indicate the degree to which trainees have changed as a result of training.

training effectiveness

Benefits that the company and the trainees receive from training

To calculate return on investment, follow these steps:

Identify outcomes (e.g., quality, accidents). Place a value on the outcomes. Determine the change in performance after eliminating other potential influences on training results. Obtain an annual amount of benefits (operational results) from training by comparing results after training to results before training (in dollars). Determine the training costs (direct costs indirect costs development costs overhead costs compensation for trainees). Calculate the total savings by subtracting the training costs from benefits (operational results). Calculate the ROI by dividing benefits (operational results) by costs. The ROI gives an estimate of the dollar return expected from each dollar invested in training.

Training outcomes or criteria

Measures that the trainer and the company use to evaluate training programs

pilot testing

Process of previewing the training program with potential trainees and managers or with other customers

Outcomes used in the evaluation of training programs

Reaction outcomes - is it collected at the program's conclusion cognitive outcomes - determine the degree to which trainees are familiar with the principles, techniques, and processes emphasized in the training program skill-based outcomes - the extent to which trainees have learned skills can be evaluated by observing their performance in work samples such as simulators affective outcomes - if trainees were asked about their attitudes on a survey, that would be considered a learning measure results: used to determine the training program's payoff for the company

A more rigorous evaluation design (pretest/posttest with comparison group) should be considered if any of the follow- ing conditions are true:

The evaluation results can be used to change the program. The training program is ongoing and has the potential to have an important influence on (employees or customers). The training program involves multiple classes and a large number of trainees. Cost justification for training is based on numerical indicators. (Here the company has a strong orientation toward evaluation.) Trainers or others in the company have the expertise (or the budget to purchase expertise from outside the company) to design and evaluate the data collected from an evaluation study. The cost of the training creates a need to show that it works. There is sufficient time for conducting an evaluation. Here, information regarding training effectiveness is not needed immediately. There is interest in measuring change (in knowledge, behavior, skill, etc.) from pre-training levels or in comparing two or more different programs.

The benefits of the training were identified by considering the objectives of the training program and the type of outcomes the program was to influence.

These outcomes included the quality of panels, housekeeping in the production area, and the accident rate.

Training cost information is important for several reasons:

To understand total expenditures for training, including direct and indirect costs. To compare the costs of alternative training programs. To evaluate the proportion of money spent on training development, administration, and evaluation as well as to compare monies spent on training for different groups of employees (exempt versus nonexempt, for example). To control costs.

Practical consideration in determine ROI

Training Program best Suited for ROI Analysis: 1. Have clearly identified outcomes 2. Are not one-time events 3. are highly visible in the company 4. are strategically focused 5. have effects that can be isolated Showing the link between training and market share gain or other higher-level strategic business outcomes can be very problematic 1. outcomes can be influences by too many other factors not directly related to training 2. Business units may not be collecting the data needed to identify the ROI of training programs 3. Measurement of training can be expensive

Determine return on investment (cont)

Utility analysis: Cost- benefit analysis method that involves assessing the dollar value of training based on: 1. estimates of the difference in job performance between trained and untrained employees 2. the number of individuals trained 3. The length of time a training program is expected to influence performance 4. the variability in job performance in the untrained group of employees

in order to identify appropriate training outcomes...

a company needs to look at its business strategy, its organizational analysis , its person analysis, its task analysis, the learning objectives of the training, and its plan for training transfer.

The pretest/posttest refers to

an evaluation design in which both pretraining and post- training outcome measures are collected

The posttest-only design refers to

an evaluation design in which only posttraining out- comes are collected.

indirect costs

are not related directly to the design, development, or delivery of the training program

Benefits

are the value that the comment gains from the training programs

Results

are used to determine the training program's payoff for the company includes increased production and reduced costs related to employee turnover, accidents, and equipment downtime as well as improvements in product quality or customer service

if the instructional objectives identified business-related outcomes such as increased customer service or product quality, then results outcomes should be included in the evaluation.

both reaction and cognitive outcomes may affect learning. Reaction outcomes provide information regarding the extent to which the trainer, facilities, or learning environment may have hindered learning. directly measure the extent to which trainees have mastered training content.

Solomon four-group design

design combines the pretest/posttest comparison group and the posttest-only control group design. In

utility analysis

is a cost-benefit analysis method that involves assessing the dollar value of training based on estimates of the difference in job performance between trained and untrained employees, the number of individuals trained, the length of time a training program is expected to influence performance, and the variability in job performance in the untrained group of employees Utility analysis requires the use of a pretest/posttest design with a comparison group to obtain an estimate of the difference in job performance for trained versus untrained employees.

Negative transfer is evident when

learning occurs but skills, affective outcomes, or results are less than at pretraining levels

Training Outcomes

or criteria refer to measures that the trainer and the company use to evaluate training programs

Success cases

refer to concrete examples of the impact of training that show how learning has led to results that the company finds worthwhile and the managers find crediible

Reaction Outcomes

refer to trainees' perceptions of the program, including the facilities, trainer, and content. they are often called class or instructor evaluations collected via a questionnaire completed by trainees.

A comparison group

refers to a group of employees who participate in the evaluation study but do not attend the training program

summative evaluation (bk)

refers to an evaluation conducted to determine the extent to which trainees have changed as a result of participating the training program involves collecting quantitative (numerical) data through test, ratings of behavior, or objective measures of performance such as volume of sales, accident, or patents

Random assignment

refers to assigning employees to the training or comparison group on the basis of chance Random assignment helps to reduce the effects of employees dropping out of the study (mortality) and differences between the training group and comparison group in ability, knowledge, skill, or other personal characteristics.

Return on Investment (ROI)

refers to comparing the trainings monetary benefits with the cost of the training referred to as level 5 evaluation. training costs can be direct and indirect

The Hawthorne effect

refers to employees in an evaluation study performing at a high level simply because of the attention they are receiving.

training effectiveness (bk)

refers to the benefits that the company and the trainees receive from training

Evaluation Design (bk)

refers to the collection of information- including what, when, how, and from whom - that will be used to determine the effectiveness of the training program

Reliability

refers to the degree to which outcomes can be measured consistently over time a reliabe test includes often for which the meaning or interpretation does not change over time

Discrimination

refers to the degree to which trainee' performance in the outcome actually reflect true difference in performance

formative evaluation ( bk)

refers to the evaluation of training that takes place during the program design and development it helps to ensure that 1. the training program is well organized and runs smoothly and 2. the trainees learn and are staffed with the program. it provides information about how to make the program better, it usually involved collecting qualitative data about the program.

criterion contamination

refers to the extent that training outcomes measure inappropriate capabilities or are affected by extraneous conditions

Criteria Relevance

refers to the extent to which training outcomes are relate dot the learned capabilities emphasized in the training program

criterion deficiency

refers to the failure to measure training outcomes that were emphasized in the training objectives

Training Evaluation

refers to the process of collecting the outcomes needed to determine whether training is effective. a training evaluation measures pecan outcomes or criteria to determine the benefits of the program

pilot testing (bk)

refers to the process of prevent the training program with potential trainees and mangers or with other customers. "dress rehearsal" to show the program manager, trainees, and customers it should be used for formative evaluation

Success cases and return on exceptions

return on expectations (ROE): process through which evaluation demonstrates to key business stakeholders that their expectations about training have been satisfied Success Cases: concrete examples of the impact of training that show how learning has led to results that company finds worthwhile

outcomes used in the Eva;uation of training programs (cont.)

return on investment: - direct costs: salaries and benefits for all employees involved in the training; program material and supplies; equipment of classroom rentals or purchases; and travel costs - indirect costs: not related directly to the design, development, or delivery of the training program - benefits: value that the company gains from the training program

A number of methods may be helpful in identifying the benefits of training:

technical, academic, and practitioner literature summarizes the benefits that have been shown to relate to a specific training program. Pilot training programs assess the benefits from a small group of trainees before a company commits more resources. Observance of successful job performers helps a company determine what successful job performers do differently than unsuccessful job performers. Trainees and their managers provide estimates of training benefits.

Statistical procedures known as analysis of variance and analysis of covariance were used to

test for differences between pretraining measures and posttraining measures for each con- dition.

There are several reasons why no evaluation or a less rigorous evaluation design may be more appropriate than a more rigorous design

that includes a comparison group, random assignment, or pretraining and posttraining measures

an evaluation design that uses only learning outcomes such as a test of knowledge of the purpose of keystones is deficient because ...

the evaluation done not measure outcomes that were included in the training objectives

taring evaluation

the process of collecting the outcomes needed to determine if training is effective

reactions have been found to have ....

the strongest relationship with post-training motivation, trainee, self-efficacy, and declarative knowledge when technology is used for instructional delivery,

There are three ways to minimize threats to validity:

the use of pretests and posttests in evaluation designs, comparison groups, and random assignment.

Time series refers

to an evaluation design in which training outcomes are collected at peri- odic intervals both before and after training.

The pretest/posttest with comparison group refers

to an evaluation design that includes trainees and a comparison group.

practicality

to the ease with which outcome measures can be collected