Quiz 5 PSY 320
Direct costs
Include salaries and benefits for all employees involved in training salaries and benefits for all employees involved in training, instructors, consultants, and employees who design the program; program material and supplies; equipment or classroom rentals or purchases and travel costs.
Behavior outcomes
Interpersonal or motor skills or behaviors acquired by learners. tests, observations, self, peer manager' ratings, work samples.
Benefits
Te value that the company gains from the training program.
Why training program should be evaluated:
1. To identify the program's strengths and weaknesses. This includes determining if the program is meeting the learning objectives, if the quality of the learning environment is satisfactory, and if the transfer of training to the job is occurring. 2. To assess whether the content, organization, and administration of the program -- Including the schedules, accommodations, trainers, and materials -- Contribute to learning and the use of training content on the job. 3. To identify which trainees benefit the most from the most or least from the program. 4. To assist in marketing programs through the collection of information from participants about whether they would recommend the program to others, why they attended the program, and their level of satisfaction in the program. 5. To determine the financial benefits and costs of the program. 6. To compare the costs and benefits of training vs non training investments (such as work redesign or better employee selection system). 7. To compare the costs and benefits of different training programs to choose the best program.
Indirect costs
Are NOT related directly to the design, development, or delivery of training program. They include general office supplies, facilities, equipment =, and related expenses; travel and expenses not directly billed to one program; training department and and staff salaries are not related to any one program and administrative and staff support salaries.
Results outcomes
Are used to determine the training program's payoff for the company. Ex of results outcome include increased production and reduced costs related to employee turnover rates of top talent (managers or other employees), accidents and equipment downtime, as well as improvements in product quality or customer service. Pay offs for the company Observation performance data from records or company databases.
Learning outcomes
Cognitive outcomes- Are used to determine the degree to which trainees are familiar w principles, facts, techniques, procedures emphasized in training program. Cognitive outcomes are LEVEL 2 (learning) criteria in Kirkpatrick's framework. Today, pencil and paper tests or self assessments are used to assess cognitive outcomes. Self-assessments refer to learner's estimates of how much they know or have learned from training. TESTS AND QUIZZES
Threats to validity
Refer to factors that will lead a evaluator to question either 1) the believability of the study results or 2. the extent to which the eval results are generalizable to other groups and trainees and situations. The believability of results refers to internal validity.
Training outcomes/criteria
Refer to measure that the trainer and the company use to evaluate training programs.
Summative eval
Refers to an evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program. That is, have trainees acquired knowledge, skills, attitudes, behavior, or other outcomes identified in training objectives? Summative evaluation usually involves collecting quantitative (numerical) data through tests, ratings of behavior, or other objective measures of performance such as volume, sales, or patents.
Resturn on investment (ROI)
Refers to comparing the training's monetary benefits w the cost of training. ROI is often referred to as level 5 evaluation.
Training effectiveness
Refers to the benefits that the company and trainees receive from training. Benefits for the company may include learning new skills and skills and behavior. Benefits for the company may include increased sales and more satisfied customers.
Evaluation design
Refers to the collection of information, including what, when, how, and from whom -- that will be used to determine the effectiveness of training program. Any organization that evaluates training, has to be confident that training that training -- rather than some other factor -- is responsible for changes in outcomes of interest (e.g., turnover, productivity). The degree of confidence that any changes in the outcomes of interest is due to training depends on the type of evaluation design that is used.
Reliability
Refers to the degree to which outcomes can be measured consistently over time. For example, a trainer gives restaurant employees a written test measuring knowledge of safety standards to evaluate safety training program that they attended. The test is given before (pertaining) and after training (post-training) employees attend the program. A reliable test includes items for which the meaning or interpretation does not change over time. A reliable test allows the trainer to have confidence that any improvements in post-training test scores from pertaining levels are the result of learning that occurred in the training program, not test characteristics (e.g, items are not understandable the second time) or the test environment (e.g., trainees performed better on the post-training test because the classroom was more comfortable and quieter.)
Discrimination
Refers to the degree to which trainees' performance on the outcome actually reflects true differences in performance. For example, a paper-and-pencil test that measures electrician's knowledge of electrical principles must detect true differences in trainees' knowledge of electrical principles. (Ppl who score high on the test have a better understanding of the principles of electricity than those who score low.)
Practicality
Refers to the ease with which the outcome measures can be collected. One reason companies give for not including learning, performance, and behavior outcomes ib their evaluation of training programs is that collecting them is too burdensome. (It takes too much time and energy, which detracts from the business.) For example, in evaluating a sales training program, it may be impractical to ask customers to rate a salesperson's behavior because this would place too much of a time commitment on the customer (and probably damage future sales relationships).
Formative evaluation
Refers to the evaluation of training that takes place during program design and development. That us formative evaluation helps ensure that 1) training program is well-organized and runs smoothly, and 2) trainees learn and are satisfied w the program. Formative evaluation provides info about how to make the program better; it usually involves collecting qualitative data about the program. Qualitative data includes: Opinions, beliefs, and feelings about the program.
Criterion contamination
Refers to the extent that training outcomes measure innaproaiate capabilities or are affected by extraneous conditions. Ex: If managers evaluations of job performance are used as a training outcome, trainees may receive may receive higher ratings of job performance simply because the managers know they attended the program, believe the program is valuable, and therefore give high ratings to ensure that the training looks like it positively affected performance. Criteria may also be contaminated if conditions under which the outcomes are measured vary from the learning environment. That is, trainees may be asked to perform their learned capabilities using equipment , time constraints, or physical working conditions that are not similar to those in the learning environment. Ex: Trainees may demonstrate spreadsheet skills using a newer version of spreadsheet software than they used in the training program. This demonstration likely will result in no changes in their spreadsheet skills from pertaining levels. I this case, poor quality training is not the cause for lack of change in their spreadsheet skills. Trainees have learned the necessary spreadsheet skills, but the environment for evaluation differs substantially from the learning environment, so no change in level is observed.
Criteria relevance
Refers to the extent which training outcomes are related to the learned capabilities emphasized in the training program. The learned capabilities required to succeed in the training program should be the same as those required to be successful on the job. The outcomes collected in training should be as similar as possible to what trainees learned in the program. One wy to ensure relevancy of the outcomes is to choose the outcomes based on the learning objectives of the program. Learning objectives show the expected action, the conditions under which the trainee is to perform, and the level or standard of performance.
Criterion deficiency
Refers to the failure to measure training outcomes that were emphasized in the training objectives. For example: the objectives of a spreadsheet skills training program emphasize that trainees both understand commands available on the spreadsheet (e.g., compute) and use the spreadsheet to calculate statistics using data set. An evaluation design that uses only learning outcomes such as a test of knowledge of the purpose of keystrokes is deficient because the evolution does not measure outcomes that were included in training objectives (e.g., use a spreadsheet to compute a mean and standard deviation of a set of data.)
Pilot testing
Refers to the process of previewing the training program w potential trainees and managers or w customers or other customers (ppl who are paying for the development of the program). Pilot testing can be used as "dress rehearsal" to show the program of managers, trainees, and customers. It should also be used for FORMATIVE EVAL.
Reaction outcomes
Refers to trainees' perceptions of the program, including facilities, trainers, and content. (Reaction outcomes are measured as a measure of "creature comfort". They are often called CLASS or INSTRUCTOR EVALUATIONS. This information is typically collected at the program's conclusion. You probably have been asked to complete class or instructor evaluations either at the end of a college course or training program at work. Reactions are either useful for identifying at what trainees thought was successful or what inhibited learning. Reaction outcomes are level 1 (reaction) criteria on Kirkpatrick's framework.
Internal validity
The internal threats to validity relate to characteristics of the company (history), the outcome measures (instrumentation, testing) and the persons in the evaluation study (maturation, regression toward the mean, mortality, initial group differences.) These characteristics can cause the evaluator to reach the wrong conclusions about training effectiveness. An evaluation study needs internal validity to provide confidence that the results of training evaluation
Training evaluation
To determine the effectiveness of training, evaluation needs to occur. Refers to the process of collecting the outcomes needed to determine whether training is effective. Process/ method for answering the question of "Did the training work?"
