Chapter 6

three dimensions characterize big data:

-volume -variety -velocity

The Evaluation Process:

1. Conduct a needs analysis 2. Develop measurable learning objectives and analyze transfer of training 3. Develop outcome measures 4. Choose an evaluation strategy 5. Plan and execute the evaluation

Workforce analytics

Practice of using quantitative methods and scientific methods to analyze data from human resource databases and other databases to influence important company metrics

Methods to control for threats to valid

Pretests and Post-test use of comparison groups random assignment

Return on expectations (ROE)

Process through which evaluation demonstrates to key business stakeholders that their expectations about training have been satisfied

Criteria relevance

The extent to which training outcomes are related to the learned capabilities emphasized in the training program

indirect costs

are not related directly to the design, development, or delivery of the training program. They include general offices supplies, facilities, equipment, and related expenses; travel and expenses not directly billed to on program.

Benefits

are the value that the company gains from the training program.

practicality

refers to the ease with which the outcome measures can be collected

reliability

refers to the degree to which outcomes can be measured consistently overtime

Why a training program should be evaluated:

1. To identify the program's strengths and weaknesses. This includes determining if the program is meeting the learning objectives, if the quality of the learning environment is satisfactory, and if transfer of training to the job is occurring. 2. To assess whether the content, organization, and administration of the program— including the schedule, accommodations, trainers, and materials—contribute to learn- ing and the use of training content on the job. 3. To identify which trainees benefit most or least from the program. 4. To assist in marketing programs through the collection of information from partici- pants about whether they would recommend the program to others, why they attended the program, and their level of satisfaction with the program. 5. To determine the financial benefits and costs of the program. 6. To compare the costs and benefits of training versus non-training investments (such as work redesign or a better employee selection system). 7. To compare the costs and benefits of different training programs to choose the best program.

Formative evaluation helps ensure that:

1. the training program is well organized and runs smoothly 2. trainees learn and are satisfied with the program

training effectiveness

Benefits that the company and the trainees receive from training

Evaluation Design

Collection of information, including whom, what, when, and how, for determining the effectiveness of the training program

Cognitive Outcomes

Determine the degree to which trainees are familiar with the principles, techniques, and processes emphasized in the training program

Summative evaluation

Refers to an evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program.

Training outcomes or criteria

Refers to measures that the trainer and the company use to evaluate training programs.

formative evaluation

Refers to the evaluation of training that takes place during program design and development.

Training Evaluation

The process of collecting the outcomes needed to determine if training is effective

Reaction Outcomes

Trainees' perceptions of the program, including the facilities, trainers, and content

Solomon four-group design

design combines the pretest/post-test comparison group and the post-test-only control group design. In the Solomon four-group design, a training group and a comparison group are measured on the outcomes both before and after training. Another training group and control group are measured only after training. This design controls for most threats to internal and external validity.

comparison group/control group

group of employees who participate in the evaluation study but do not attend the training program

direct costs

includes salaries and benefits for all employees involved in training, including trainees, instructors, consultants, and employees who design the program; program material and supplies; equipment or classroom rentals or purchases; and travel costs

utility analysis

is a cost-benefit analysis method that involves assessing the dollar value of training based on estimates of the difference in job performance between trained and untrained employees, the number of individuals trained, the length of time a training program is expected to influence performance, and the variability in job performance in the untrained group of employees.

affective outcomes

outcomes including attitudes and motivation

dashboard

refers to a computer interface designed to receive and analyze the data from departments within the company to provide information to managers and other decision-makers.

Time series

refers to an evaluation design in which training outcomes are collected at periodic intervals both before and after training. (In the other evaluation designs discussed here, training outcomes are collected only once after and maybe once before training.) The strength of this design can be improved by using reversal, which refers to a time period in which participants no longer receive the training intervention.

Return on Investment (ROI)

refers to comparing the training's monetary benefits with the cost of the training.

Big Data

refers to complex data sets developed by compiling data across different orga- nizational systems including marketing and sales, human resources, finance, accounting, customer service, and operations.

Hawthorne effect

refers to employees in an evaluation study performing at a high-level simply because of the attention they are recieving

Threats to validity

refers to factors that will lead an evaluator to question either 1. the believability of the study results or 2. the extent to which the evaluation results are generalizable to other groups of trainees and situation

self-assessments

refers to learners estimates of how much they know or have learned from the training

criterion contamination

refers to the extent that training outcomes measure inappropriate capabili- ties or are affected by extraneous conditions. For example, if managers' evaluations of job performance are used as a training outcome, trainees may receive higher ratings of job performance simply because the managers know they attended the training program, believe the program is valuable, and therefore give high ratings to ensure that the train- ing looks like it positively affects performance.

criterion deficiency

refers to the failure to measure training outcomes that were emphasized in the training objectives.

pilot testing

refers to the process of previewing the training program with potential trainees and managers or with other customers (persons who are paying for the development of the program). Pilot testing can be used as a "dress rehearsal" to show the program to managers, trainees, and customers. It should also be used for formative evaluation.

internal validity

relate to characteristics of the company (history), the outcome measures (instrumentation, testing), and the persons in the evaluation study (maturation, regression toward the mean, mortality, initial group differences)

external validity

the extent to which the results of a study can be generalized to other situations and to other people

cost-benefit analysis

the process of determining the economic benefits of a training program using accounting methods that look at training costs and benefits

Discrimination

to the degree to which trainees' performance on the outcome actually reflects true differences in performance

Skill-based outcomes

used to assess the level of technical or motor skills and behaviors