Chapter 6: Validity
Construct
(latent variables) are scientifically developed concepts, ideas, or hypotheses that are used to describe or explain behavior
table of specifications
(test blueprint) is a two-dimensional chart that guides instrument development by listing the content areas that will be covered on a test, as well as the number (proportion) of tasks or items to be allocated to each content area
manifest (or observed) variables
A construct is defined or inferred by a group of interrelated variables or dimensions that can be directly measured or observed, which are known as ___________________________.
uncontaminated
A criterion measure should be ______________________, meaning that the measure should not be influenced by any external factors that are unrelated or irrelevant to the criterion.
confirmatory factor analysis
A procedure in which researchers, using factor analysis, consider the theory associated with a test and propose a set of underlying factors that they expect the test to contain; they then conduct a factor analysis to see whether the factors they proposed do indeed exist.
fluid intelligence and crystallized intelligence
According to Horn and Cattell, intelligence encompasses two general types, and these are:
the predictor variable, the criterion, the criterion measure
An aptitude test (_____) could be designed to predict job performance (_____) as measured by supervisor ratings (_____).
measuring the wrong thing or may be used in inappropriate ways.
Assessment results may be highly reliable, but may be
reliable
Because an instrument is judged based on its relationship to a criterion, the criterion itself needs to be measured accurately; it must be ___________________.
directly or observed
Constructs cannot be measured _______________ or _________________.
response processes
Evidence based on _______________________, sometimes viewed as part of content validity, provides validity support that examinees used intended cognitive and psychological processes during test taking.
validity coefficients
Evidence of relations in other variables is evaluated by _____________________, which indicate the relationship between test results and a criterion measure.
-factors within the test: ambiguous, inappropriate, or poorly constructed test items; too few test items; improper arrangement of items; identifiable patterns of answers -administration and scoring procedures: failure to follow standard directions and time limits, giving test takers unfair aid, scoring errors -test-taker characteristics: emotional disturbances, test anxiety, lack of motivation -inappropriate test group: giving the test to individuals who differ from the validation group
Other factors that can impact the validity of the interpretation of test results:
test content, response process, internal structure, relations to other variables, and testing consequences
Sources of information upon which an argument for the validity of an instrument can be made
(1) Clearly delineate the construct or the content domain to be measured. (2) Develop a table of specifications (3) The actual test items are written (4) Outside consultants are recruited to review the test items to determine if they represent the content domain
Steps in the content validation process
test validation
The process of __________________________ involves gathering and evaluating multiple sources of validity evidence that support the interpretations and inferences made from test scores.
relevant
To determine if a criterion measure is __________________ to the intended purpose of the test, one must ascertain whether it is a valid representation of the behavior or outcome in question.
construct underrepresentation and construct-irrelevant variance
Two main threats to validity
1. Validity refers to the appropriateness of the use and interpretation of test results, not the test itself. 2. Validity is a matter of degree; it does not exist on an all-or-none basis. 3. Validity is a unitary concept. It is a single, unified concept that centers on construct validity, which is supported by sources of validity evidence. 4. Validity is specific to some particular group, use, or purpose of the test score interpretations--no test is valid for all purposes or all groups.
When using the term validity in relation to tests and measurement, consider:
criterion
a direct and independent measure that the test is designed to predict or be correlated with
exploratory factor analysis
a method of factor analysis in which researchers do not propose a formal hypothesis but instead use the procedure to broadly identify underlying components
Factor analysis
a statistical procedure that analyzes the relationships among test components to determine whether a test is unidimensional
predictor variable
a variable whose scores are used to estimate scores of a criterion variable
true negative
correctly predicting failure
true positive
correctly predicting success
Age differentiation studies
focus on demonstrating validity by showing the degree to which test scores increase or decrease with age
Construct underrepresentation
implies that a test is too narrow and fails to include important dimensions or aspects of the identified construct
Criterion measures
indexes designed to assess or measure the particular criterion
Criterion-related evidence
involves examining the relationships between test results and external variables that are thought to be a direct measure of the construct
discriminant validity evidence
is based on consistently low correlations between the test and other tests that are supposed to measure different constructs
false negative error
is falsely predicting that a negative outcome would occur and it does not
false positive error
is falsely predicting that a positive outcome would occur and it does not
evidence of convergent validity
is obtained by correlating instrument results with the results of other instruments that assess the same construct
Construct-irrelevant variance
means that the instrument is too broad and contains too many variables, many of which are irrelevant to the construct
false negative
predicting failure but having a positive or successive outcome
false positive
predicting success but having a negative or failing outcome
Crystallized intelligence
refers to acquired knowledge and ability obtained through education and personal experience continues to increase through middle age and maintains during old age
Content validity evidence
refers to evidence of the relationship between a test's content and the construct it is intended to measure.
evidence based on consequences of testing
refers to examining the intended and unintended consequences of the use of test results
test homogeneity
refers to how uniform the items and components of a test are in measuring a single concept this is one means of providing evidence of the internal structure of a test
response processes
refers to the actions, thought processes, and emotional traits that the test taker invokes in responding to a test
Validity
refers to the adequacy and appropriateness of the uses of assessment results
Concurrent validity
refers to the degree to which a predictor variable is related to some criterion at the same time (concurrently)
Fluid intelligence
the ability to solve problems and adapt to new situations improves through childhood and adolescence, then begins to decline as early as late adolescence and into adulthood
predictive validity
the degree to which a test score estimates some future level of performance
Validity
the degree to which all the accumulated evidence supports the intended interpretation of test scores for the proposed purpose
construct validity
the extent to which test results can be interpreted in terms of a particular construct
a false positive error or a false negative error
two types of prediction errors
exploratory and confirmatory
two ways to conduct factor analysis
Group differentiation (contrasted group) studies
used to analyze whether an instrument differentiates between different groups of individuals in ways predicted by theory
