Stats Quiz: Reliability & Validity

Reliability

Reliability refers to the consistency of a measure. Psychologists consider three types of consistency: over time (test-retest reliability), across items (internal consistency), and across different researchers (interrater reliability).

Internal Consistency

The consistency of people's responses across the items on a multiple-item measure. All the items on such measures are supposed to reflect the same underlying construct, so people's scores on those items should be correlated with each other •e.g., on the Rosenberg Self-Esteem Scale, people who agree that they are a person of worth should tend to agree that that they have a number of good qualities How to assess? •Cronbach's α (value of +.80 or greater is generally taken to indicate good internal consistency

Face Validity

The extent to which a measurement method appears "on its face" to measure the construct of interest. • e.g., most people would expect a self-esteem questionnaire to include items about whether they see themselves as a person of worth and whether they think they have good qualities. How to assess? • having a large sample of people rate a measure in terms of whether it appears to measure what it is intended to—it is usually assessed informally.

Interrater Reliability

The extent to which different observers are consistent in their judgments • e.g., measuring college students' social skills, you could make video recordings of them as they interacted with another student whom they are meeting for the first time. Then you could have two or more observers watch the videos and rate each student's level of social skills. How to assess? •Cronbach's α (value of +.80 or greater is generally taken to indicate good internal consistency

Content Validity

The extent to which people's scores on a measure are correlated with other variables (known as criteria) that one would expect them to be correlated with. • e.g., people's scores on a new measure of test anxiety should be negatively correlated with their performance on an important school exam. • Or imagine that a researcher develops a new measure of physical risk taking. People's scores on this measure should be correlated with their participation in "extreme" activities such as snowboarding and rock climbing How to assess? • requires collecting data using the measure

Discriminant Validity

The extent to which scores on a measure are not correlated with measures of variables that are conceptually distinct. • e.g., For example, self-esteem is a general attitude toward the self that is fairly stable over time. It is not the same as mood, which is how good or bad one happens to be feeling right now. So people's scores on a new measure of self-esteem should not be very highly correlated with their moods

Validity

Validity is the extent to which the scores from a measure represent the variable they are intended to

Test-Retest Reliability

When researchers measure a construct that they assume to be consistent across time, then the scores they obtain should also be consistent across time. • e.g., intelligence is thought to be consistent across time (highly intelligent today, highly intelligent next week How to assess? • Using the same measure on a group of people at one time, using it again on the same group of people at a later time •Graphing the data in a scatterplot and computing Pearson's r •High test-retest correlations = consistency over time