PSYO 373 - Reading 2: Construct Validation

How will the inter item correlation matrix differ for uni- and multi-dimensional pools of items?

• One-dimensional pools should all correlate together • Multi-dimensional pools will have some items that correlate together and others that do not.

Why should initial item pools be over-inclusive?

• Pool should be broader and more comprehensive than the theoretical model of the construct • Pool should include some items that will be shown to be unrelated to the target construct • Important for establishing the boundaries of the construct • You can always take items away but you can't add items after

Why is a literature review the important first step in scale construction?

• A lit review will reveal if a psychometrically sound measure already exists • Also helps to develop which construct and which parts of it you will be measuring

When is it typically ("often") considered and when should it be considered?

• Construct validity should be considered throughout the process of constructing a measure • It is typically only considered at the end

Describe convergent, discriminant, concurrent, and predictive validity.

• Convergent validity: the extent to which a measure correlates with other measures of the same construct • Discriminant validity: the extent that the measure does not correlate with measures of other constructs that are theoretically distinct • Concurrent validity: relating a measure to criterion evidence collected at the same time as the measure itself • Predictive validity: associations with criteria that are assessed at some point in the future

What are the pros and cons of dichotomous response formats?

• Dichotomous items are often less reliable and scales must be longer in order to reach same reliability • They are easier to score and analyze • They take less time to complete

Describe the features of good and bad candidate items

• Good candidate items load at least moderately with one primary factor and minimally with other factors • Bad items load weakly on a hypothesized factor of cross-load on more than one factor

What is the difference between internal consistency and homogeneity?

• Internal consistency is the overall degree of inter-relation among a set of items o Alpha coefficient • Homogeneity is the extent to which all of the items on a given scale tap a single factor o Must look at mean and distribution of inter-item correlations

Why is it useful to write a formal definition of a construct in the very early stages of test development?

• It can address where the gaps in your knowledge are and force you to research the areas you do not quite understand • Finalize one's model of the construct and clarify its breadth and scope

Why is it not a static quality of a test that can be established with a single study?

• It is a dynamic process • It is a principle for making inferences rather than a characteristic of the test • A test yields many inferences but only some of them are valid

What is construct validity and how is it related to other forms of validity?

• It is a process rather than an endpoint • It should be consulted at all points of developing a measure, not just at the end • It is best understood as an over arching concept of all types of validity • Construct validity encompasses all forms of validity and subsumes all the major types of reliability

What is examined in the external validity phase and how is the task different from the task in the structural validity phase?

• Looks at convergent and discriminant validity and criterion-related validity phase • Examines whether the relations between the new measure and important test and non-test criteria are congruent with one's theoretical understanding and the nomological net

Contrast the rational-theoretical and criterion-keying methods of item selection and mention the shortcomings with each method.

• RT Method: write items that appear consistent with particular theoretical understanding o Must assume that theoretical model is entirely correct o Poor discriminant validity • CK Method: Items selected solely on their ability to discriminate between the two groups o Doesn't matter if they are relevant of face valid o Fail to advance theory in meaningful way o Highly heterogeneous

What is factor analysis and why is it a useful tool in scale construction?

• Reducing the matrix of inter-item correlations to a set of factors or components that can be used to form provisional scales • Develops homogeneous scales with good discriminant validity

For the multi trait-multi method matrix on p. 255, why is the PV-NV correlation of -.29 a problem, especially with regards to the PV-PV correlation of .15?

• That is a rather high correlation for two items that are distinct especially when PV should correlate higher that .15 with itself

What is content validity?

• The degree to which elements of an assessment instrument are relevant to and representative of the targeted construct for a particular assessment purpose

Why should items with extreme endorsement probabilities not be automatically dumped?

• You want to be equally precise across all levels of the continuum