Chapter 7
What is a conservative change estimate for the MCID?
-A conservative estimate is a change of 0.5SD -If someone changes by .5 SD that is probably a significant improvement
What is the Minimum Detectible Difference?
-A lot like CI, but applied to interpretations of change, not interpretations of a single score. -The amount of change that is not due to error alone -The amount of change in a variable that must be achieved to reflect a true difference; the smallest amount of change that passes the threshold of error.
Another way of looking at associations with other variables is by looking at discriminant validity. What is it?
-A test should correlate weakly with constructs that are theoretically different. (you don't' want everything to correlate equally, they should correlate based on degree of how related). -The degree to which an instrument yields different results when measuring two different constructs. -Ex: A measure of height, pants inseam and IQ. Height correlate with pants inseam more than with IQ. -Ex: The results of an IQ test should not be expected to correlate with results of a gross motor test.
What is criterion-related validity?
-Ability of one test to predict results obtained on an external criterion test. -This form of validity hinges on the presence of a golden standard -The strongest criteria(gold standards) are behavioral and observable. For example, the ability of the SAT to predict first-semester college grades(observable). -There are two subtypes of criterion-related validity: Predictive and concurrent
Validity of change issues: Baseline Scores
-Also affect the validity of change scores -Sometimes the extent to which a variable will change depends on its starting point. -Perhaps because of the measure, e.g., floor or ceiling effects restrict the ability of the measure to show change. If all of the scores are clustered at the top of the measure, there is nowhere to improve. -Perhaps a treatment works better for more impaired people than for less impaired people, or vice versa.
What is the difference between concurrent an convergent validity?
-Concurrent (sub-type of criterion-related validity) versus convergent (subtype of construct validity): highly highly similar. -Difference is that convergent validity is not necessarily being related to a golden standard.
Another way of looking at associations with other variables is by looking at convergent validity. What is it?
-Convergent validity: the degree to which two different instruments or methods are able to measure the same construct. -Ex: If two health status scales are valid methods of measuring quality of life, they should produce similar scores. -Might sound similar to concurrent validity.
What is the Minimum potentially detectible difference?
-Depends on the precision of the measuring tool being used -This instrument has only whole numbers, nothing in the "middle", from 1-6 -Ex: There is not 5.5
Construct validity: What is internal structure?
-Do components of the test work together in the way that you think they do
What is content validity/logical validity?
-Does the test cover everything being tested and nothing else -Does the test provide a representative and exhaustive assessment of the content domain under consideration, and nothing irrelevant? -Ex: If giving an aphasia test does it cover all aspects of aphasia and nothing that is not aphasia. If an so-called aphasia test fails to assess repetition, it has questionable content validity.
What are "other" forms of validity?
-Ecological validity -Validity of change scores
What is ecological validity?
-Ecology(real world) -How well the test content simulates or relates to "real world" contexts -This term applies more to the validity of experimental studies than it does to clinical testing. -It indicates people's reactions to how well the test content simulates or relates to "real world" contexts. Or does the test feel artificial? -For example, some people might say that the Texas Functional Living Scale is more ecologically valid than cognitive testing in predicting ability to live independently since it has things like writing a check or looking in the phonebook to find a number.
What are the different types of validity?
-Face validity -Content validity -Criterion-related validity -Construct validity
True/false: True change can be measured using interval and ratio scores.
-False -True change can only be measured using ratio scores because only ratio scores measure the absolute amount of something --Interval scores will tell you the distance of change, but not the true amount of change
True/false: Validity is not a matter of degree.
-False -Validity is a matter of degree. -Things fall on a spectrum of how valid they are for a particular use
What is Face Validity?
-Indicates that an instrument appears to test what it's supposed to and that is a plausible method for doing so. -Does the test looks like it's doing it's job -Ex: If giving test of artic and the test looks like it's testing artic, then that's face validity
What is the formula for Minimum Detectible difference?
-Like formula for CI besides the root of 2. -MDD95 = 1.96* SEM √(2) -MDD95 means that 95% of stable patients demonstrate a random variation of less than the MDD95 range
What is an example of MDD? What does MDD95 mean?
-MDD95 means that 95% of stable patients demonstrate a random variation of less than the MDD95 range -If the observed score = 100 and the MDD95 is 5, it means that 95% of stable(aren't changing) patients demonstrate random variation between 95 and 105. -So if have a person who scores 100 on first time and 103 on second time, that is in that range of normal variation and probably not a significant difference that could be accounted for random error alone. -If have someone that scores 100 on first time and 106 on the second time, that's outside the 95% MDD so that's more likely due to a real change. Probably not explained just to error alone.
Can you ask is this instrument valid or is the MOCA a valid instrument?
-No -Instead say how valid is this instrument for a given purpose?"
Validity of change issues: Level of measurement
-Nominal data cannot be subtracted and cannot demonstrate change because it is just a category (like numbers on someone's jersey) -True change can only be measured using ratio scores because only ratio scores measure the absolute amount of something --Interval scores will tell you the distance of change, but not the true amount of change
What is an important purpose of this test's application?
-One main purpose is to maintain the test taker's motivation. -No examinee wants to complete a test that looks absurd and irrelevant.
Construct validity: Associations with other variables. What is Known group validity?
-One way of doing this is known groups validity: a test with good construct validity should be able to discriminate among people with who are known to have the trait and those that do not. -If give a new test of language to kids with already-diagnosed language disorders, these kids should score lower on a new test of language than kids who do not have language disorders. -One way of validating the test since if know who is good and bad already, and give a new test, then should be able to predict who will perform good or badly on test.
Criterion related validity subtypes: What is predictive validity?
-Predict a future criterion score, typically 6 months or later. -Ex: SAT scores are supposed to predict Freshman year GPA. -Important for assessing future risk, e.g., development of reading problems.
What are some examples of a diagnostic test?
-Test accuracy against a gold standard: sensitivity, specificity, PPV, NPV, LR+, LR-
Validity of change issues: Stability
-That stability of the construct measure also affect the validity of change scores -If the construct itself changes often, it's hard to make strong inferences about the effect of a treatment. --Blood pressure, pain, etc. (things that vary often, it is harder to make stronger conclusions about the change scores). -Determination of true change may require taking several measurements both at baseline and outcome to demonstrate that the change in score is stable.
What is validity of change? What are the issues that affect it?
-The difference between the outcome and the initial score is called a change score or a difference score. -Ex: If pretest = 80, post = 88, change score = +8 -Issues affecting validity of change --Level of measurement --Reliability --Stability --Baseline Scores
What is validity?
-The extent to which an instrument measure what it is intended to measure. -Can you make appropriate inferences from test scores?
Construct validity: Response processes
-The match between the psychological processes that respondents actually use when completing a measure and the processes that the test developer intended. -For example, sometimes people forget the instructions or use some idiosyncratic approach to completing a test.
Construct validity: Consequences of use
-This gets back to validity as being evaluated within a context and for a specific context --In other words, the social consequences of test interpretation are a facet of validity -Ex: For example, if a test were used to form inappropriate conclusions. Tests conducted on psychopaths to determine if they would get parole or not. Results were interpreted all wrong and no one got parole, even people that deserved it because the test was interpreted poorly. If consequences of use are wrong and how can affect construct validity of the test.
Answer from previous question
-This shows poor reliability with a big (wide) SEM -Scores are so wide that it's hard to know if meaningful since error is so influential. -This is why want to use measures with good reliability so that if there is a small change you have the power to detect that change.
True/false: Face validity is the least rigorous method of evaluating validity.
-True -Face validity is the least rigorous method of evaluating validity. Jokingly called "armchair validity."
True/false: Construct validity builds on other forms of validity. Other things have to be true in order for a test to have good construct validity.
-True -If not good content validity then not good construct validity. -If not good internal structure, then not good construct validity.
True/false: Distance on an ordinal scale between intervals is not known and may not be equal.
-True -Ordinal data are often misinterpreted -Distance between intervals is not known and may not be equal. -Ordinal "numbers" have no arithmetic properties. The sum of ordinal data and change scores based on this sum are mathematically meaningless. --THIS IS A MAJOR PROBLEM WITH NOMS (distance with numbers may not be equal (going from a 4-5 may not be the same as 6-7).
True/false: Validity is for a given purpose. An instrument/test is not universally valid or invalid.
-True -You wouldn't say "this hammer is an invalid tool" -A hammer is neither valid nor invalid, it's the way you use the hammer --Appropriate (valid) for hammering in nails and pulling out nails, inappropriate (invalid) for sawing wood.
What is Minimal Clinically Important Difference (MCID)?
-What level of change has to occur for that to be perceptible and important to the patient. -Once it is determined that a change in score is not just due to error, it is helpful to determine whether that change is noticeable and important to patients. -Ex: If improve from 50 to 53 on the WAB, that might exceed the error of measurement, but it may be completely undetectable by patients(patient doesn't feel different) or their family, and may not confer any "real world" benefit so then who cares. If not detectable or meaningful then doesn't exceed this criterion for MCID.
Criterion related validity subtypes: What is concurrent validity?
-When a test to be validated and criterion when both are obtained at the same time. -Often used to establish the validity of new screening or diagnostic tests. -Ex: When someone develops a new test, they probably gave it at the same time next to another test to see how well it does next to a gold standard.
For validity, measurements are important insofar as they allow us to do what?
Measurements are important insofar as they allow us to make generalizations beyond a specific score.
What is construct validity?
The ability of an instrument to measure an abstract concept, or construct, that is not directly observable(not real).
Answer from previous question
This shows good reliability since it has a small (narrow) SEM
True/false: Validity describes the use of the test and not the test itself.
True
Validity of change issues: Reliability
We've already covered this when we talked about measurement error and minimal detectable difference. A change in obtained score could simply reflect error of the measurement tool. --That's why it's good to have a measurement tool with a high level of reliability