Psyc Testing chpt 7: Utility
Utility Tests (utility analysis)
- Some are straightforward, while others are more sophisticated, employing complicated mathematical models -Often address the question of "which test gives us the most bang for the buck?"
The Cut Score in Use
-Cut scores may be RELATIVE, in which case they are determined in reference to normative data (e.g. selecting people in the top 10% of test scores). -FIXED cut scores are made on the basis of having achieved a minimum level of proficiency on a test (e.g. a driving license exam).
IRT Based Methods
-In an IRT framework, each item is associated with a particular level of difficulty. -In order to "pass" the test, the testtaker must answer items that are deemed to be above some minimum level of difficulty, which is determined by experts and serves as the cut score. -makes use of the item-mapping method and bookmark method
Methods of Setting Cut Scores
-The Angoff Method -The Known Groups Method -IRT Based Methods -Method of Predictive Yield -Discriminant Analysis
Utility Analysis Includes...
-expectancy data -Taylor-Russell tables: validity, selection ratio, and base rate -Naylor-Shrine tables -Brogden-Cronbach-Gleser formula -utility gain
The Angoff Method of Setting Cut Scores
-judgments of experts are averaged to yield cut scores for the test -can be used for personnel selection, traits, attributes, and abilities -problems arise if there is low agreement between experts
Factors Affecting Utility
-psychometric soundness -costs (including economic and non-economic) -benefits
Practical Considerations
-the pool of job applicants -complexity of the job -the cut score in use (relative or fixed) -multiple cut scores -multiple hurdles
Limitations of Taylor-Russell Tables
-the relationship between the predictor (the test) and the criterion (rating of performance on the job) must be linear -the potential difficulty of identifying a criterion score that separates "successful" from "unsuccessful" employees
Decision Theory and Utility (Cronbach and Gleser)
Cronbach and Gleser (1965) presented: 1) a classification of decision problems; 2) various selection strategies ranging from single-stage processes to sequential analyses; 3) quantitative analysis of the relationship between test utility, the selection ratio, cost of the testing program, and expected value of the outcome; and 4) a recommendation that in some instances job requirements be tailored to the applicant's ability instead of the other way around (adaptive treatment).
Economic Costs of Utility
Economic costs may include purchasing a test, a supply bank of test protocols, and computerized test processing. -Other economic costs are more difficult to calculate such as the cost of not testing or testing with an inadequate instrument
Endpoint of Utility Analysis
Endpoint of a utility analysis yields an educated decision as to which of several alternative courses of action is most optimal (in terms of costs and benefits).
Other Variables in Utility Analysis
Many other variables may play a role in selection decisions besides test results, including applicants' minority status, general physical or mental health, or drug use.
Multiple Hurdles
Multiple hurdles - achievement of a particular cut score on one test is necessary in order to advance to the next stage of evaluation in the selection process (e.g. Miss America contest).
Costs of Utility
One of the most basic elements of a utility analysis is the financial cost associated with a test. - Cost in the context of test utility refers to disadvantages, losses, or expenses in both economic and noneconomic terms.
Psychometric Soundness Factor of Utility
Psychometric soundness - Generally, the higher the criterion validity of a test the greater the utility. -There are exceptions because many factors affect the utility of an instrument and utility is assessed in many different ways. -valid tests aren't always useful tests
Method of Predictive Yield of Setting Cut Scores
R. L. Thorndike (1949) proposed this norm-referenced method -Took into account the number of positions to be filled, projections regarding the likelihood of offer acceptance, and the distribution of applicant scores.
The Pool of Job Applicants
Some utility models are based on the assumption that for a particular position there is a limitless pool of candidates. -However, some jobs require such expertise or sacrifice that the pool of qualified candidates may be very small. -The economic climate also affects the size of the pool. -The top performers on a selection test may not accept a job offer.
Taylor-Russell Tables of Utility Analysis
Taylor- Russell tables provide an estimate of the percentage of employees hired by the use of a particular test who will be successful at their jobs, given different combinations of three variables: the test's validity, the selection ratio used, and the base rate.
The Complexity of the Job
The same utility models are used for a variety of positions, yet the more complex the job the bigger the difference in people who perform well or poorly.
Non-economic Benefits of Utility
may include a better work environment and improved morale
Selection-Ratio of the Taylor-Russell Tables
refers to a numerical value that reflects the relationship between the number of people to be hired and the number of people available to be hired
Utility Gain in the Brogden-Cronbach-Gleser Formula
refers to an estimate of the benefit monetary or otherwise) of using a particular test or selection method
Base Rate of the Taylor-Russell Tables
refers to the percentage of people hired under the existing system for a particular position
Validity of the Taylor-Russell Tables
refers to the validity coefficient
Naylor-Shrine Tables of Utility Analysis
tables entail obtaining the difference between the means of the selected and unselected groups to derive an index of what the test (or some other tool of assessment) is adding to already established procedures. -For both Taylor-Russell and Naylor-Shine tables the validity coefficient comes from concurrent validation procedures.
Utility
the usefulness or practical value of testing to improve efficiency
Brogden-Cronbach-Gleser Formula of Utility Analysis
used to calculate the dollar amount of a utility gain resulting from the use of a particular selection instrument under specified conditions.
Utility Analysis
a family of techniques that entail a cost-benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment.
Multiple Cut Scores
The use of multiple cut scores for a single predictor (e.g. students may achieve grades of A, B, C, D, or E).
Benefits of Utility
We should take into account whether the benefits of testing justify the costs of administering, scoring, and interpreting the test. -benefits can be defined as profits, gains, or advantages -successful testing programs can yield higher work productivity and profits for a company
Discriminant Analysis of Setting Cut Scores
a family of statistical techniques used to shed light on the relationship between identified variables (such as scores on a battery of tests) and two (and in some cases more) naturally occurring groups (such as persons judged to be successful at a job and persons judged unsuccessful at a job).
Potential Benefits of Utility
an increase in the quality of workers' performance; an increase in the quantity of workers' performance; a decrease in the time needed to train workers; a reduction in the number of accidents; a reduction in worker turnover.
The Known Groups Method of Setting Cut Scores
entails collection of data on the predictor of interest from groups known to possess, and not to possess, a trait, attribute, or ability of interest -after analysis of a data, a cut score is chosen that best discriminates the groups -one problem with known groups method is that no standard set of guidelines exist to establish guildelines
Non-economic Costs of Utility
include things such as human life and safety
Expectancy Data of Utility Analysis
likelihood that a testtaker will score within some interval of scores on a criterion measure
