Week 4 Assignment Notes

People processing strategies

Actions that organizations use when socializing newcomers.

Cost-benefit analysis

Comparing the monetary costs of training to the benefits received in nonmonetary terms, such as improvements in attitudes, safety, and health.

5. Archival Performance Data

Use of existing information, such as files or reports

Change and acquisition

The last stage in the new employee socialization stage, which occurs when new employees accept the norms and values of the group or organization, master the tasks they must perform, and resolve any role conflicts and overloads.

Organizational socialization

The process of adjusting to a new organization. It is a learning process whereby newcomers must learn a wide variety of information and behaviors to be accepted as an organizational insider.

HRD evaluation

The systematic collection of descriptive and judgmental information necessary to make effective training decisions related to the selection, adoption, value, and modification of various instructional or HRD activities.

Pretest and post-test

including both a pretest and a post-test allows the trainer to see what has changed after the training. If the majority of trainees already know the material covered in training before they start it, then high scores on the post-test measure of learning become much less impressive.

At least three types of data are available for evaluating HRD effectiveness:

individual performance systemwide performance and economic.

A realistic job preview (RJP)

involves providing recruits with complete information about a job and an organization.

According to Nickols, a stakeholder is

"a person or group with an interest in seeing an endeavor succeed and without whose support the endeavor would fail."

Nickols recommends the use of a

"stakeholder scorecard" for training evaluation.

In general, successful socialization means that the newcomer develops

(1) greater knowledge of the organization and work group; (2) attitudes that make performing, fitting into, and remaining with the organization and work group possible; and (3) behaviors that lead to personal and organizational effectiveness.

insiders possess three essential elements:

1. Accurate Expectations—insiders normally know what to expect of the situations in which they find themselves, so there are fewer surprises to confront them. Newcomers' expectations are more likely to differ from organizational reality. 2. Knowledge Base—when surprises do occur, insiders have the knowledge base (from history and experience in the setting) to more accurately make sense of the surprising event. Newcomers generally lack this knowledge. 3. Other Insiders—insiders have coworkers with whom to compare their judgments and interpretations of organizational events. Newcomers have not yet developed the relationships with insiders they can trust and draw upon to help them interpret organizational events.

Drawing upon techniques of capital budgeting, the three phases of Cascio's approach are as follows:

1. Compute the minimum annual benefits required to break even on the program (e.g., how much of a payback must the program generate in order to cover its costs?) 2. Use break-even analysis to determine the minimum effect size (dt) that will yield the minimum required annual benefit (how much of an improvement in job performance must the trained employees show for the program to generate the payback needed to break even?) 3. Use the results from meta-analytic studies to determine the expected effect size and expected payoff from the program (what is the likely degree of improvement in job performance that the HRD program being proposed has shown in previously conducted research on this program or method?)

Other common questions that are used on questionnaires include:

1. Do you think that this session was worthwhile? 2. How well did the trainer(s) do the job? 3. What did you like most about this session? 4. What did you like least, and how could we improve? 5. Do you have any comments or suggestions?

Brinkerhoff extends the training evaluation model to six stages. He suggests a cycle of overlapping steps, with problems identified in one step possibly caused by negative occurrences in previous steps. His stages or steps are:

1. Goal Setting: What is the need? 2. Program Design: What will work to meet the need? 3. Program Implementation: Is the design working (with focus on implementation of the program)? 4. Immediate Outcomes: Did participants learn? 5. Intermediate or Usage Outcomes: Are participants using what they learned? 6. Impacts and Worth: Did the program make a worthwhile difference to the organization?

Three additional reasons for conducting HRD evaluations are:

1. If HRD staff cannot substantiate its contribution to an organization, its funding and programs may be cut during the budgeting process, especially if an organization faces tough times. 2. Evaluation can build credibility with top managers and others in an organization. 3. Senior management often wants to know the benefits of HRD programs (see the Opening Case on LensCrafters).

Edgar Schein describes three dimensions of organizational roles. These are:

1. Inclusionary—a social dimension (e.g., outsider, probationary status, permanent status) 2. Functional—a task dimension (e.g., sales, engineering, plant operations) 3. Hierarchical—a rank dimension (e.g., line employee, supervisor, middle manager, officer)

However, two serious problems can occur when relying on self-report data:

1. Mono-method bias. If both reports in a before-and-after evaluation come from the same person at the same time (say, after training), conclusions may be questionable. The respondents may be more concerned about being consistent in their answers than about providing accurate responses. 2. Socially desirable responses. Respondents may report what they think the researcher (or boss) wants to hear, rather than the truth. For example, employees may be fearful or embarrassed to admit that they learned nothing in a training program.

Given the many constraints placed on HRD efforts, Grove and Ostroff recommend the following:

1. Perform a needs analysis 2. Develop an explicit evaluation strategy 3. Insist on specific training objectives 4. Obtain participant reactions 5. Develop criterion instruments (to measure valued outcomes) 6. Plan and execute the evaluation study

Cynthia Fisher divides the content of socialization into five categories of learning:

1. Preliminary learning—including the discovery that learning will be necessary, what to learn, and whom to learn from 2. Learning about the organization—including its goals, values, and policies 3. Learning to function in the work group—including the values, norms, roles, and friendships within it 4. Learning how to perform the job—including the necessary skills and knowledge for a particular job 5. Personal learning—learning from experience with the job and organization, including self-identity, expectations, self-image, and motivation

Other important supervisor orientation functions include:

1. Providing (or arranging for) training in job specifics 2. Buffering the newcomer from demands outside the work group for a period of time to facilitate job learning 3. Providing challenging initial assignments 4. Conducting timely, constructive performance evaluations 5. Diagnosing problems at work that create conflicts 6. Using the newcomer's arrival as an opportunity to reallocate tasks or redesign work to improve effectiveness and employee satisfaction with the work system

Putting this all together, our Expanded Kirkpatrick Evaluation Framework is as follows:

1. Reaction a. Perceived usefulness/utility—what was the perceived relevance/usefulness of this training? (this can help with modifying training content and design) b. Post-training attitudes—how well did trainees like the training? Did it impact trainee self-efficacy? 2. Cognitive learning—how much did trainees learn from the training? a. Posttraining learning—how much learning does the trainee demonstrate immediately after training? b. Retention—how much learning does the trainee demonstrate back on the job? (most often assessed some months later, via retesting) 3. Behavior—what behavior change occurred as a result of training? a. Training performance—how well can trainees demonstrate the newly acquired skills at the end of training? b. Transfer performance—how well can trainees demonstrate the newly acquired skills back on the job? (most often assessed some months later, for example via supervisory evaluation or performance tests) 4. Results a. What tangible outcomes or results occurred as a result of training? b. What was the return on investment (ROI) for this training? (see ROI and utility sections below; this is Phillips' fifth level) c. What was the contribution of this training program to the community/larger society?

William Horton discussed how an online or e-learning course might be evaluated using Kirkpatrick's four levels:

1. Reaction—easy to gather continuous feedback online; could use a discussion thread or chat room to allow trainees to discuss their experiences with online learning 2. Learning—very easy to test trainees electronically (at least for objective tests); can also link to a learning management system that reports results to trainees and stores these results for organizational purposes 3. Behavior—very hard to capture electronically, although some relevant data may be available in other information systems, for example, appraisals, promotions, turnover, and discipline data could be tapped from an organization's Human Resource Information System (HRIS), or Learning Management System (LMS) 4. Results—even harder to do online than traditionally, without face-to-face interaction, feedback, and buy-in; conservative estimates of online course impact should be used to build credibility

The perceptual nature of organizational roles can lead to the following three situations:

1. Role overload—when the employee perceives the role as being more than he or she can reasonably do 2. Role conflict—when the employee receives mixed messages about what is expected of him or her by others, such as a boss and coworkers 3. Role ambiguity—when the employee feels the role is unclear; this is often the result of assuming a newly created position

Corning follows ten steps in designing its program:

1. Set objectives; e.g., Corning's objectives included: a. Reduce turnover in the first three years of employment by 17 percent b. Reduce time to learn the job by 17 percent c. Provide newcomers with a uniform understanding of the organization d. Build a positive attitude toward the organization and communities 2. Form a steering committee 3. Research orientation as a concept 4. Interview recently hired employees, supervisors, and corporate officers 5. Survey the orientation practices of top companies (the program at Texas Instruments proved particularly helpful) 6. Survey existing organizational orientation programs and materials 7. Select content and delivery method 8. Pilot and revise materials 9. Produce and package print and audiovisual materials 10. Train supervisors and install the system

Goldstein suggests that efforts at training evaluation have moved through four stages since the 1960s:

1. Stage One focuses on anecdotal reactions from trainers and program participants. Judging from the survey results cited earlier, it appears many organizations still operate at this level. 2. Stage Two involves borrowing experimental methodology from academic laboratories to use for program evaluation. Organizational constraints (including time, resources, and the inability to randomly select participants or use control groups that receive no training) make application of these designs difficult, thus discouraging evaluation efforts. 3. Stage Three creatively matches the appropriate research methodology to existing organizational constraints, thus making program evaluation more practical and feasible. 4. Stage Four recognizes that the entire training and HRD process affects the organization and shifts the focus of evaluation from postprogram results to the entire HRD process.

A model of the RJP process (see Figure 8-2) suggests four interrelated mechanisms: vaccination, self-selection, coping, and personal commitment.

1. Vaccination Against Unrealistically High Expectations 2. Self-Selection 3. Coping Effect 4. Personal Commitment

Kraiger, Ford, & Salas (1993)

A classification scheme that specifies three categories of learning outcomes (cognitive, skill-based, affective) suggested by the literature and proposes evaluation measures appropriate for each category of outcomes

Confidentiality

A commitment not to disclose information regarding an individual's performance in training, or other personal matters

Utility analysis

A computation that measures in dollar terms the effect of an HRD program in terms of a change in some aspect of the trainee's performance.

Informed consent

A form stating that the participants in an evaluation study have been informed of these facts and agree to participate in the study.

Validity

A vital issue to consider when selecting a data collection method. Validity is concerned with whether the data collection method actually measures what it is intended to measure, that is, are we hitting the right target?

Return on investment (ROI)

A measure of the benefit the organization receives by conducting the training program. It is the ratio of the results divided by the training costs.

Research design

A plan for conducting an evaluation study.

Random assignment

A process in training program design and evaluation whereby the trainer randomly assigns individuals to the training and control groups. This increases the researcher's confidence that any differences between the training and the control condition were brought about by the training, and not some factor that differed between individuals in the two conditions.

Meta-analysis

A research technique where the researchers combine the results from many other studies on a given topic to look for average effects or correlations between variables. Meta-analyses can sometimes overcome the problems of small sample sizes that often are experienced by HRD evaluation studies.

Role

A set of behaviors expected of individuals who hold a given position in a group. Roles define how a person fits into the organization and what he or she must do to perform effectively.

Hazing

A situation where new employees are targets of practical jokes or are harassed because they lack certain information.

2. Questionnaire

A standardized set of questions intended to assess opinions, observations, and beliefs

Encounter

A step in the new employee socialization process that begins when a recruit makes a formal commitment to join the organization by either signing an employment contract or simply accepting an offer of employment or membership. At this point, an individual crosses the inclusionary boundary separating the organization from the outside environment and begins to discover what the organization is really like. During this stage, employment expectations may be confirmed or rejected.

Practicality

A vital issue to consider when selecting a data collection method. Practicality is concerned with how much time, money, and resources are available for the evaluation method.

Brinkerhoff (1987)

Six stages: Goal Setting, Program Design, Program Implementation, Immediate Outcomes, Intermediate or Usage Outcomes, and Impacts and Worth

Information overload

The idea that a person can absorb only so much information in a given period before learning efficiency drops and stress increases.

1. Interview

Conversation with one or more individuals to assess their opinions, observations, and beliefs

Self-report data

Data provided directly by individuals involved in the training program. It is the most commonly used type of data in HRD evaluation, but also has serious weaknesses in terms of the potential value for HRD evaluation.

The following issues should be considered when determining RJP content:

Descriptive or Judgmental Content—descriptive content focuses on factual information, while judgmental content communicates incumbents' feelings. Extensive or Intensive Content—extensive content contains all pertinent information, while intensive content implies selective information that is presented more briefly and forcefully. Degree of Content Negativity—should the content of the RJP be highly negative, moderately negative, or somewhere in between? Message Source—if an audiovisual medium is used, should actors, job incumbents, or other organization members, such as supervisors or trainers, present the message?

According to Phillips, evaluation can help to:

Determine whether a program is accomplishing its objectives Identify the strengths and weaknesses of HRD programs, which can lead to changes, as needed Determine the cost-benefit ratio of an HRD program Decide who should participate in future HRD programs Identify which participants benefited the most or least from the program Gather data to assist in marketing future programs Establish a database to assist management in making decision

KIRKPATRICK Learning (Level 2)

Did the trainees learn what the HRD objectives said they should learn? This is an important criterion that an effective HRD program should satisfy. Measuring whether someone has learned something in training may involve a quiz or test—clearly a different method from assessing the participants' reaction to the program.

KIRKPATRICK Reaction (Level 1)

Did the trainees like the program and feel it was valuable? At this level, the focus is on the trainees' perceptions about a program and its effectiveness. This is useful information. Positive reactions to a training program may make it easier to encourage employees to attend future programs. But if trainees did not like the program or think they didn't learn anything (even if they did), they may discourage others from attending and be reluctant to use the skills or knowledge obtained in the program. The main limitation of evaluating HRD programs at the reaction level is that this information cannot indicate whether the program met its objectives beyond ensuring participant satisfaction.

To calculate the cost of a training program, an HRD professional should consider five categories of expenses.

Direct costs, Indirect costs, Development costs, Overhead costs, Compensation for participants

KIRKPATRICK Behavior (Level 3)

Does the trainee use what was learned in training back on the job? This relates back to our discussion of training transfer in Chapter 3. This is also a critical measure of training success. We all know coworkers who have learned how to do something but choose not to. If learning does not transfer to the job, the training effort cannot have an impact on employee or organizational effectiveness. Measuring whether training has transferred to the job requires observation of the trainee's on-the-job behavior or viewing organizational records (e.g., reduced customer complaints, a reduction in scrap rate).

Several authors suggest modifications to Kirkpatrick's four-level approach that keep the framework essentially intact. These include:

Expanding the reaction level to include assessing the participants' reaction to the training methods and efficiency Distinguishing more clearly between cognitive and affective reactions to training Splitting the reaction level to include assessing participants' perceptions of enjoyment, usefulness (utility), and the difficulty of the program Adding a fifth level (beyond results) to specifically address the organization's return on investment (ROI) Adding a fifth level (beyond results) to address the societal contributions and outcomes created by an HRD program Adding a focus on "return on expectations" (ROE), i.e., focusing more on what managers and executives want and expect from the HRD intervention, rather than on ROI.

Why aren't evaluations done better, and more frequently?

First, conducting an evaluation is not an easy process. It requires time, resources, and expertise that the HRD staff may not have or may not be willing to expend. Second, many factors beyond the program itself (including the economy, equipment, policies and procedures, other HR efforts, and resource availability) can affect whether employee performance improves, thus making it difficult to evaluate the impact of training. Third, those associated with HRD programs may be afraid of criticism and program cuts if the evaluation shows that the program is not effective.

Phillips (1996)

Five levels: Reaction and Planned Action, Learning, Applied Learning on the Job, Business Results, Return on Investment

Kirkpatrick (1967, 1987, 1994)

Four levels: Reaction, Learning, Job Behavior, and Results

CIPP (Galvin, 1983)

Four variables: Context, Input, Process, and Product

KIRKPATRICK Results (Level 4)

Has the training or HRD effort improved the organization's effectiveness? Is the organization more efficient, more profitable, or better able to serve its clients or customers as a result of the training program? Meeting this criterion is considered the bottom line as far as most managers are concerned. It is also the most challenging level to assess, given that many things beyond employee performance can affect organizational performance. Typically at this level, economic and operating data (such as sales or waste) are collected and analyzed.

Holton (1996)

Identifies five categories of variables and the relationships among them: Secondary Influences, Motivation Elements, Environmental Elements, Outcomes, Ability/Enabling Elements

Expectation

In the context of new employee socialization, an expectation is a belief about the likelihood that something will occur and can encompass behaviors, feelings, policies, and attitudes.

76 percent of HR leaders say employee onboarding practices are underutilized at their organization.

Kronos

3. Direct Observation

Observing a task or set of tasks as they are performed and recording what is seen

the following practices should be included in a research design:

Pretest and post-test Control group

Realistic job preview (RJP)

Providing recruits with complete information about the job and the organization (including both positive and negative information).

4. Tests and Simulations

Structured situations to assess an individual's knowledge or proficiency to perform some task or behavior

Time series design

The collection of data over time, which allows the trainer to observe patterns in individual performance.

Reliability

The consistency of results from a test or evaluation measure. Reliability concerns freedom from error and bias in a data collection method. A method that has little or no error or bias is highly reliable, whereas the results of a method that has significant error or bias is unreliable and cannot be trusted.

Role orientation

The extent to which individuals are innovative in interpreting their organizational roles, versus "custodial" in maintaining what has been done in that role previously.

Cost-effectiveness analysis

The financial benefits accrued from training, such as increases in quality and profits, or reduction in waste and processing time.

Anticipatory socialization

The first stage of the new employee socialization process, which begins before the individual joins the organization. In this stage, the person forms an impression about what membership in an organization is like from a variety of sources, such as rumors, anecdotes, advertisements, the media, and recruiters.

Advocates suggest a number of conditions in which an RJP can be both useful and effective, including:

When job candidates can be selective about job offers, especially during times of low unemployment • When the selection ratio is low (the organization has many more job applicants than positions available) • When the recruits are unlikely to have enough information available to them to develop realistic expectations (such as with entry level, complex, or unique jobs) • When replacement costs are high

Control group

a control group is a group of employees similar to those who receive training, but they don't receive training at the same time. However, this group receives the same evaluation measures as the group that is trained, which allows for a comparison of scores. The ideal scenario occurs when the training group and the control group have similar scores before training, and then the scores for the training group increase after training, while those of the control group remain constant. This provides fairly strong evidence that the training (and not some other factor) is responsible for the changes on the outcome measures.

4. Personal Commitment

a recruit who makes a decision to join an organization based on a realistic perspective will likely develop a stronger personal commitment to that choice. This encourages job satisfaction and a long-term commitment to remain with an organization.

First, when conducting an evaluation, both _____________________________ may be collected.

descriptive and judgement information

response shift bias

in which respondents' perspectives of their skills before training change during the training program and affect their after-training assessment.

Finally, evaluation is conducted to help managers, employees, and HRD professionals

make informed decisions about particular programs and methods.

Descriptive information

provides a picture of what is happening or has happened, whereas judgmental information communicates some opinion or belief about what has happened.

1. Vaccination Against Unrealistically High Expectations

providing accurate information to outsiders is similar to vaccinating people against a disease. Recruits are given information that permits them to adjust their expectations to the reality of the job. For example, a realistic portrayal of typical overtime or weekend work may assist applicants in understanding all that will be required of them in a given job.

2. Self-Selection

realistic expectations enable recruits to decide whether the job and the organization match their individual needs. If they are incompatible, the recruit will probably not accept the position, thus saving the organization from hiring someone who would likely be dissatisfied and quit. The model suggests that self-selecting individuals are more likely to be satisfied employees. Self-selection obviously assumes that an organization has enough applicants that it can afford to let them select themselves out of the hiring process.

3. Coping Effect

realistic expectations help newcomers develop a clear idea of their roles, which in turn enables them to develop coping strategies for performing their jobs effectively.

norms

rules of conduct (typically unwritten) that are established by group members to influence or control behavior within a group.

judgmental information

someone's opinion based on the facts.

Second, evaluation involves the ______________________________ of information according to a predetermined plan to ensure the information is appropriate and useful.

systematic collection

effectiveness is determined with respect to

the achievement of a goal or a set of goals.

HRD effectiveness must be determined in relation to

the goals of the program or programs being examined.