Psychometrics Final- Validity

Content validity

A process in which we look at aspects of the test itself to determine whether or not there is reasonable overlap between the test and construct of interest. -or- The relationship between the content of a test and some well-defined domain of knowledge or behavior. How well the content of the test taps into the construct. A good match between content and domain=high content validity

How do you assess construct validity?

Factor Analysis 1st time is exploratory, 2nd time is confirmatory (Sample size has to be very large) -Correlation more often used because experimental manipulation is unethical

Is it correct to say a test is valid?

False Measurement is considered valid when the test measures what it is supposed to. Validity is a matter of degree

Why is the predictive validation strategy considered the ideal approach? What is this approach not always utilized?

It is ideal because we get the whole picture; it minimizes range restriction. It is not always utilized because it is expensive, and unethical and usually large numbers are involved.

3 steps in assessing content validity:

It is the degree to which the content of a measure truly reflects the full domain of the construct for which it is being used, no more and no less. 1. describe the content domain in terms of boundaries and structure 2. determine the areas of the content domains (boundaries) that are measured by each test item 3. compare the structure of the test with the structure of the content domain

Strengths and weakness of predictive validation

Strengths: Accurate Minimizes range restriction Best way to screen out failures Weaknesses: Not efficient Not cost effective Unethical

Construct Underrepresentation

What wasn't covered by the test that is relevant to the construct (this happens when we don't fully understand the construct) the construct is not adequately represented by the test's measurement -decreases validity

What is the impact of large base rate?

When base rate is large most everyone would be successful, but there are a limited number of positions, so there will be a high number of true positives and a high number of false negatives *large base rate: .7-.9

What is the impact of a small base rate?

When base rate is small, hardly anyone would be successful, but must fill number of positions, so there will be a high number of true negatives and a high number of false positives

What types of information can be gained from the use of multitrait-multimethod matrix approach?

-potential method bias -convergent validity/discriminate validity Convergent Validity-monotrait-teromethod relationships to determine convergent validity (across diagonal in slides). ---if the correlations are all significant, we can say we have evidence of convergent validity. Discriminant Validity-heterotrait-heteromethod relationships looking at different traits across different method relationships (dashed triangles in slides). -----Looking for non significant relationships Method Bias-lookign at monomethod heterotrait relationships. Different traits, within the same method (sold triangles in slide). ---Hope to find non significant relationships.

What are three elements of the MTMM?

1.) Multitrait- measures different traits 2.) Multimethod- uses various methods to measure those traits 3.) Matrix- a table comprised of correlations between the methods and traits measured

What are the possible decision outcomes that were presented in class? Which of these outcomes are we looking to maximize?

1.) True positive (successful performance and accept individual)accept someone who is a good fit 2.) False Positive (failure performance and accept individual) Accept someone who fails 3.) True Negative (Successful performance and reject individual) reject someone who would fail 4.) False negative (failure performance and accept reject individual) reject someone who is a good fit ***Maximize true positive and True Negative

High content validity=

A good match between content and domain (construct)

What is range restriction?

A restriction of who is able to participate from the population Reduces correlation between predictor (test score) and criterion (outcome measure) Can be direct through selection by the predictor; can be indirect through decisions based on the criterion. Goal is often to screen out failures-allowed for in predictive validation not concurrent validation.

strengths and weaknesses of the concurrent validation

Advantages: Practical Easier Usually correlates in same ballpark as predictive Not as morally iffy Weaknesses: increases range restriction reduces correlation cannot screen out failures

Validity

An issue of tapping into the construct that it is supposed to. tied to psychological theory and to the implications of test scores. Validity is a matter of degree.

Valid measurement

Appropriate measurement used for what we are testing. The amount of overlap between the construct and the test Ideally, the test and construct would totally overlap and that would be a completely valid measurement.

Difference between base rate and selection ratio

Base rate: level of performance on the criterion in the general population. (e.g., 75% of population is successful,.the base rate is .75). Success of being a brain surgeon would have a very low base rate. -Selection ratio: ratio of positions to applicants (e.g., if 30 people apply for 3 jobs, the selection ratio=10% or .10) -Both allow us to determine the likelihood of a successful decision beyond pure randomness

Distinguish between boundaries and structure of a domain

Boundaries: an understanding of all aspects/ facets that are operating within the construct. Cover all aspects. (Breadth) Structure: The relative weight or importance of each of the factors/facet within the construct. (depth

How does predictive validation procedure differ in concurrent validation procedure?

Concurrent: you obtain test scores and criterion scores from a preselected population at the same time. Predictive: obtain test scores and criterion scores at different times. You also use all participants who are available.

How does content validity and construct validity differ?

Content validity: looks at a single test and compares aspects within the test together. -theory- Construct validity: compares different tests to one another -empirical-

What is the difference between convergent and discriminant validity?

Convergent validity (form of construct validity): we expect to find significant correlations between our measures and other measures that purport to tap into the construct of interest. -Not all constructs have been normed -Negative correlation is still convergent validity Discriminant validity: behaviors not associated with our construct of interest should be uncorrelated with our test. -There will always be relationship in some way, we want low relationship with unrelated constructs.

What would you expect from convergent and discriminant validity in terms of the resulting correlation analysis?

Convergent: significant correlations Discriminant- should have low correlations That there will be some overlap because there is some level of correlation between constructs.

Heterotrait-Monomethod

Different Trait/Same method -Method Bias- -Solid Triangles-

Heterotrait Heteromethod correlation

Different measures different traits -Discriminant Validity- -Dashed Triangles-

Why is it good to have a measure that appears to be valid, but not at the expense of empirical validity?

Empirical validity trumps face validity. Empirical validity must come first * just because a test looks like it is measuring what it says it's measuring, doesn't mean that it does.

What are the 3 methods of determining construct validation?

FEC Factor Analysis- Analyzing which group of items "hang" together, and whether these clusters look like the construct conceptually. We do this to see how well we fit the boundaries Experimental Manipulation- Manipulate the construct of interest (i.e., induce fear) and see if it relates to different scores on our test. This method is not feasible Correlational Study- correlations between measures of certain behaviors (that are either related to or unrelated to our construct of interest) and our test.

How does one conduct a predictive validation procedure?

First, obtain test scores (job application) Second, you obtain performance measures and correlate these with the test scores.

What is face validity?

If a test looks like it measures its target construct Can have big impact on construct relevant variance. -If someone knowing what you're measuring will not effect them, go with high face validity -if knowing will effect them (such as racism) (low social desirability), then use low face validity -Don't confuse this with an empirical approach

What level of selection ratio has the biggest impact on correct decision making?

If selection ratio is low(# of positions and # of applicants almost equal), doesn't really matter what method of prediction you use because you're taking almost everyone who applies -Validity has the biggest impact on correct decision making when selection ratio is low.

Why is an assessment of content validity seen less in psychological measurement than in educational testing?

In education there is a set answer (IQ is easier to measure), but in our line of work psychology the concepts are more ambiguous. It's harder to pin down the boundaries and structure.

How does increased or decreased validity play into the relationship between construct underrepresentation, valid measurement, and construct irrelevant variance?

Increased validity means that the scores are more likely to be true decreased validity means that the test scores are less likely to be accurate Increases or decreases in the amount of overlap between the two increases or decreases the degree to which the measure is valid. The degree to which the instrument maps onto the boundaries of the construct (i.e.,missing facets of construct with instrument increases construct underrepresentation)

How can range restriction impact our validity for decisions?

It decreases validity, so it is weaker The more range restriction, the more deviant the scores are between predictive and concurrent validation.

Monotrait-Heteromethod

Same trait-different method -Convergent Validity- Diagonal

Construct Irrelevant variance

Test content that reflects issues, characteristics or concepts that are irrelevant to the construct for which a test is to be interpreted. Factors other than the construct that the test picks up on (things beyond the test or construct) Extraneous influence (systematic variation and random error) Decreases validity and brings in noise Ex. bad mood

What level of base rate is ideal in terms of accurate decision making?

Tests used as predictors are most likely to have an impact on accurate decision making when the base rate is moderate (.50)

Reliability

The consistency of measurement. It is a relatively simple quantitative property of test responses

Construct Validity

The degree to which a test measures what it claims, or purports, to be measured. The relationship between our instrument and another instrument that looks at the same construct. -use factor analysis to assess construct validity- -Factor analysis shows which items cluster together conceptually (inter-relationship) -Does my test measure the abstract concept I think it measures?

What is method bias?

The factors or issues related to the administration of the test that may affect the validity of the test, ex: social desirability Method bias is discovered by looking at the same method across different traits relationship. A single method that cannot distinguish between traits --if significant correlation within one method, then rater needs training or test is poor assessment OR constructs are significantly related (i.e., conceptually overalp) -If multiple traits are significantly correlated in one method then method bias **two methods are enough to determine the D and C validity

What is a criterion?

a measure that could be used to determine the accuracy of a decision.

What is criterion-related validity

association of test scores with meausres of success. 2 ways to assess criterion-related validity: 1.) predictive validity 2.) concurrent validity

Monotrait-monomethod

same construct same method reflects reliability-the correlation of a measure with itself.

How would one determine the variance in the criterion that can be explained by the predictor measure?

square the reliability coefficient