Reliability and Validity of Measures

Criterion-Related Validity

-Ability of one test to predict results obtained by external gold standard tool 3 Factors to Determine This: 1: Strength of that gold standard 2: The administration of 2 tests should be independent of one another 3: The two tools need to measure the same thing.

Construct Validity

-Ability of the tool to measure an abstract construct or context (something you can observe) -The challenge is when its an abstract construct (can't be observed)

Responsiveness to Change

-Change and Difference Scores: difference between baseline and outcome 4 Factors influence the Validity of This: 1: Level of Measurement: must be numerical, or else we can't measure it 2: Reliability: does it measure what it's supposed to 3: Stability: if the trait we're measuring is prone to vary, then the difference between the scores is suspect 4: Baseline scores: Only measuring the narrow band might not show true scores (regression to the mean)

Face Validity

-Does the instrument appear to measure what it's supposed to measure? -Weakest form of validity, but still important -Important because the more it looks like it tests the right thing, the more clients/people will believe it

Rater Reliability

-Examines the testers 1: Intrarater: stability of the data recorded by one individual across 2 or more trials 2: Interrater: variation between 2 or more raters who would measure the same person at the same time under the same circumstances NOTE: must first have good intrarater reliability before you can have good interrater reliability

Regression to the Mean

-If we start out with an extreme score, it will automatically regress to the mean as we test it more. -Makes it difficult to know if they are getting better/worse, or simply regressing to the mean.

Test-Retest Reliability

-Looks at stability of the tool -Does the tool measure the variable consistently -Think about other factors that may effect a 2nd testing (i.e. the test actually helps them learn, so the 2nd test they do better because they have been exposed to it before.)

Alternate Form Reliability

-More than one form/test to prevent people from sharing answers -Must make sure both are equivalent -Typically for pen and paper forms -Look at correlation between the 2 forms

3 Causes of Systematic Error

-Rater/tester: the tester's fault -Instrument: the instrument was faulty -Variability of trait: issues with the actual thing you are measuring.

Internal Consistency/Homogeneity Reliability

-Reflects extent to which the items measure various aspects of the same characteristic or trait, and nothing else. -The scale/measure is grounded in a theory that defines its dimensions

Content Validity

-The adequacy in which the universe is sampled by a test -I.E does it cover the theoretical construct limits? How adequately have we covered what we should be covering? Usually determined by a panel of experts -Also looks at bias

Prediction

-Validity of Measures -Attempt to tell us/predict about basic function based on a score or measure. Note: not all assessments are predictive

Evaluation

-Validity of Measures -Extent or magnitude of characteristic or condition (how much impairment)

Discrimination

-Validity of Measures -Help us to separate among different groups of individuals (i.e. people who need OT versus people who don't need OT)

Types of Reliability

1: Test-retest 2: Rater (intrarater and interrater) 3: Alternative forms 4: Internal consistency

Chrombach's Alpha

Correlates all items to each other, and if the items are summative, correlates them to the total score. Items that don't correlate you would get rid of.

Reliability

The extent to which a measurement is consistent and free from error, and whether it can be reproduced and bring the same results.

Validity

Whether or not it measures what it says it measures