Chapter 5: Validity

Evaluating Validity Coefficients

- Look for changes in the cause of relationships: The logic of criterion validation presumes that the causes of the relationship between the test and the criterion will still exist when the test is in use. - For example, a test might be used and shown to be valid for selecting supervisors in industry; however, the validity study may have been done at a time when all the employees were men, making the test valid for selecting supervisors for male employees.

Be sure that sample size was adequate

Another problem to look for is a validity coefficient that is based on a small number of cases. - Sometimes a proper validity study cannot be done because there are too few people to study. - A common practice is to do a small validity study with the people available. - The smaller the sample, the more likely chance variation in the data will affect the correlation. - A good validity study will present some evidence for cross validation. - A cross validation study assesses how well the test actually forecasts performance for an independent group of subjects.

Review the Subject Population in the Validity Study

Another reason to be cautious of validity coefficients is that the validity study might have been done on a population that does not represent the group to which inferences will be made.

Concurrent Validity Evidence

Concurrent-related evidence for validity comes from assessments of the simultaneous relationship between the test and the criterion. - Concurrent evidence for validity applies when the test and the criterion can be measured at the same time. - E.g.; learning disability test and school performance. - Here the measures and criterion measures are taken at the same time because the test is designed to explain why the person is now having difficulty in school.

Content-Related Evidence for Validity

Content-related evidence for validity of a test or measure considers the adequacy of representation of the conceptual domain the test is designed to cover. - Content validity evidence has been of greatest concern in educational testing. - For example, if you are being tested on the first six chapters of this book, then content-related evidence of validity is provided by the correspondence between the items on the test and the information in the chapters.

Criterion-Related Evidence for Validity

Criterion validity evidence tells us just how well a test corresponds with a particular criterion. - Such evidence is provided by high correlations between a test and a well-defined criterion measure. - The reason for gathering criterion validity evidence is that the test or measure is to serve as a "stand-in" for the measure we are really interested in.

Review Evidence for Validity Generalization

Criterion-related validity evidence obtained in one situation may not be generalized to other similar situations. - Generalizability refers to the evidence that the findings obtained in one situation can be generalized—that is, applied to other situations.

What does the Criterion mean?

Criterion-related validity studies mean nothing at all unless the criterion is valid and reliable. - Some test constructors attempt to correlate their tests with other tests that have unknown validity. - A meaningless group of items that correlates well with another meaningless group remains meaningless.

Discriminant Evidence

Discriminant evidence indicates that the measure does not represent a construct other than the one for which it was devised. - To demonstrate discriminant evidence for validity, a test should have low correlations with measures of unrelated constructs, or evidence for what the test does not measure.

Construct-Related Evidence for Validity

Established through a series of activities in which a researcher simultaneously defines some construct and develops the instrumentation to measure it. - This process is required when "no criterion or universe of content is accepted as entirely adequate to define the quality to be measured". - Construct validation involves assembling evidence about what a test means.

Face Validity

Face validity is the mere appearance that a measure has validity. - We often say a test has face validity if the items seem to be reasonably related to the perceived purpose of the test. - Face validity is really not validity at all because it does not offer evidence to support conclusions drawn from test scores.

Predictive Validity Evidence

The forecasting function of tests is actually a type or form of criterion validity evidence known as predictive validity evidence. - For example, the SAT Critical Reading Test serves as predictive validity evidence as a college admissions test if it accurately forecasts how well high-school students will do in their college studies. - The SAT, including its quantitative and verbal subtests, is the predictor variable, and the college grade point average (GPA) is the criterion.

Validity Coefficient

The relationship between a test and a criterion is usually expressed as a correlation called a validity coefficient. - This coefficient tells the extent to which the test is valid for making statements about the criterion. - In practice, one rarely sees a validity coefficient larger than .60, and validity coefficients in the range of .30 to .40 are commonly considered high. - A coefficient is statistically significant if the chances of obtaining its value by chance alone are quite small: usually less than 5 in 100. - The validity coefficient squared is the percentage of variation in the criterion that we can expect to know in advance because of our knowledge.

Defining Validity

Validity can be defined as the agreement between a test score or measure and the quality it is believed to measure. - Validity is sometimes defined as the answer to the question, "Does the test measure what it is supposed to measure?" - Validity is the evidence for inferences made about a test score. There are three types of evidence: 1. construct-related 2. criterion-related 3. content-related - The most recent standards emphasize that validity is a unitary concept that represents all of the evidence that supports the intended interpretation of a measure.

Convergent Evidence

When a measure correlates well with other tests believed to measure the same con- struct, convergent evidence for validity is obtained. - This sort of evidence shows that measures of the same construct converge, or narrow in, on the same thing. - Convergent evidence is obtained in one of two ways: 1. We show that a test measures the same things as other tests used for the same purpose. 2. We demonstrate specific relationships that we can expect if the test is really doing its job.