Training and Development: CHAPTER 6

Return on Investment

--Direct Costs: salaries and benefits for all employees involved in training, program material and supplies; equipment or classroom rentals or purchases; and travel costs --Indirect Costs: not related directly to the design, development, or delivery of the training program --Benefits: value that the company gains from the training program

Determining Return on Investment

-Cost-Benefit analysis: process of determining the economic benefits of a training program using accounting methods that look at training costs and benefits -ROI should be limited only certain training programs, because it can be costly

Threat to Validity

-Factors that will lead an evaluator to question either -internal or external validity

Training Outcomes or Criteria

-Measures that the trainer and the company use to evaluate training programs

Methods to Control for Threats to Validity

-Pretests and post-tests: comparison of the post training and pre training measures can indicate degree trainees have changed as result of training -hawthorne effect -random assignment

Random Assignment

-assigning employees to the training or comparison group on basis of chance alone - it is often impractical -analysis of covariance

Training effectiveness

-benefits that the company and the trainees receive training

Evaluation Design

-collection of information, including whom, what, when, and how for determining the effectiveness of the training program

Solomon Four-Group

-combines the pretest/post-test comparison group and the post-test only control group design ==this design controls for most threats to internal and external validity

Reasons for Evaluating Training

-companies make large investments in training and education and view them as a strategy to be successful; they expect the outcomes of training to be measurable -training evaluation provides the data provides the data needed to demonstrate the training does provide benefits to the company (formative and summative evaluation)

Q: which of the following statements is true of comparison groups

-comparison groups consists of a group of employees who do not attend the training program

Training Quality Index (TQI)

-computer application that collects data about training department performance, productivity, budget, and courses, and allows detailed analysis of data (quality is included in effectiveness)

Dashboards

-computer interface designed to receive and analyze the data from departments within the company to provide information to managers and other decision makers = USEFUL because they can provide a visual display using charts of the relationship between learning activities and business performance data

Success Cases

-concrete examples of the impact of training that show how learning has led to results that the company finds worthwhile

Q: The evaluation process ideally begins with

-conducting a needs analysis

Utility Analysis

-cost-benefit analysis method that involves assessing the dollar value of training based on: -estimates of the difference in job performance between trained and untrained -number of individuals trained -length of time program is expected to influence performance -variability in job performance in untrained group of employees

Q: If a manager believes that a training program is valuable and hence provides higher ratings of job performance to those trainees who attended the training program, then the training outcomes will lack relevance due to __________

-criterion contamination

Relevance

-criterion contamination= extent that training outcomes measure inappropriate capabilities or are affected by extraneous condition

Reliability

-degree to which outcomes can be measured consistently over time

Discrimination

-degree to which trainees' performance on the outcomes actually reflects true differences in performance

Cognitive Outcomes

-determine the degree to which trainees are familiar with the principles, techniques, and processes emphasized in the training program

Summative Evaluation

-determines the extent to which trainees have changed as a result of participating in the training program =may include measuring the monetary benefits that company receives from the program (ROI) -involves quantitative data

Q: If trainers are interested in the generalizability of a study's results to other group and situations, then they are said to be interested in the ______ of study

-external validity

Q: ROI analysis is best suited for training programs that are

-focused on an operational issue

training programs best suited for ROI analysis

-have clearly identified outcomes -are not one-time events -are highly visible in company -are strategically focused -have effects that can be isolated

To calculate ROI

-identify outcomes -place a value on the outcomes -determine the change in performance after eliminating other potential influences on training results -obtain annual amount benefits -determine training costs -calculate ROI by dividing operational results by costs (ROI gives estimate of dollar return expected from each $ invested in training)

Summative Evaluation: training program should be evaluated to...

-identify the program's strengths and weaknesses -to assess whether content, organization, and administration of the program contribute to learning and the use of training content on job -identify which trainees benefited most or least from program -gather data to assist in marketing training programs -determine financial benefits and costs -compare costs benefits of (training vs. non-training investments and different training programs) -a training program

Affective Outcomes

-if trainees were asked about their attitudes on a survey, that would be considered a learning measure

Pretest/Post Test with Comparison Group

-includes trainees and a comparison group ==differences between each of the training conditions and the comparison group are analyzed determining whether differences between the groups were caused by training

Reaction Outcomes

-it is collected at the program's conclusion

Evaluation Practices

-it is important to recognize the limitations of choosing to measure only reaction and cognitive outcomes =to ensure an adequate training evaluation, companies must collect outcome measures related to both learning and transfer

Determining Costs

-methods for comparing costs of alternative training programs include the resource requirements model and accounting

Q: typically _________ are used to assess cognitive outcomes

-observations

Post-Test Only

-only post-training outcomes are collected ==appropriate when trainees can be expected to have similar levels of knowledge, behavior, or results outcomes prior to training

Criterion Deficiency

-outcomes identified by needs assessment and included in training objectives

Showing link between training and market share gain or other higher-level strategic business outcomes can be problematic

-outcomes influenced too many other factors not directly related -business units not collecting data needed to identify ROI training programs -measurement of training can be expensive

Contamination

-outcomes measured in evaluation

Relevance outcomes

-outcomes related to objectives

Types of Evaluation Designs

-post-test only -pretest/post-test -pretest/post-test with comparison group -time series -solomon four group

Workforce Analytics

-practice using quantitative methods and scientific methods to analyze data from human resource database and other databases to influence important company metrics

Pretest/Post Test

-pre training and post training outcome measures are collected ==used by companies that want to evaluate a training program but are uncomfortable with excluding certain employees

Pilot Testing

-process of previewing the training program with potential trainees and managers or with other customers

Return on Expectation (ROE)

-process through which evaluation demonstrates key business stakeholders that their expectations about training have been satisfied

American Society of Training Society and Development (ASTD)

-provides information about training hours and delivery methods that companies use to benchmark

Q: according to kirkpatrick's framework for categorizing training outcomes _____ are level 1 criteria that are often called class or instructor evaluations

-reaction

Outcomes used in the Evaluation of Training Programs

-reaction outcomes -cognitive outcomes -skill-based outcomes -affective outcomes -results -return on investment -training quality index (TQI)

Formative Evaluation

-takes place during program design and development =helps ensure that training program is well organized and runs smoothly =trainees learn and are satisfied with program -provides information about how to make program better - collecting qualitative data -pilot testing

Determining benefits-methods include

-technical, academic, and practitioner literature -pilot training programs and observance of successful job performers -estimates by trainees and their managers

Internal Validity

-the believability of the study results

Practicality

-the ease with which the outcome measures can be collected

External Validity

-the extent to which the evaluation results are generalizable to other groups of trainees and situations

Skill-Based Outcomes

-the extent to which trainees have learned skills can be evaluated by observing their performance in work samples (simulators)

Criteria

-the extent to which training outcomes are related to the learned capabilities emphasized in the training program

Training Evaluation

-the process of collecting the outcomes needed to determine if training is effective

Q: Formative evaluation involves collecting qualitative data about a training program from trainees mainly through

-their opinions and feelings about the program

Reversal

-time period in which participants no longer receive the training intervention

Time Series

-training outcomes are collected at periodic intervals both before and after training ==it allows an analysis of the stability of training outcomes over time

Hawthorne Effect

-use of comparison groups: group of employees who participate in evaluation study but do not attend training program

Results

-used to determine the training program's payoff for the company

Levels of Kirkpatrick

1. reaction 2. learning 3. activity 4. results

The Evaluation Process

conduct a needs analysis > develop measurable learning objectives and analyze transfer training > develop outcome measures > choose an evaluation strategy > plan and execute the evaluation