OT Assessment Exam 1: Lecture Content

How to control variables leading to error

-Be trained in administering and interpretation of test -Recheck that correct procedures are being used -Be aware of biases, natural tendencies -Be aware of the influences of ones background, presence, or expectations -Be unobtrusive, and aware of test takers variable that might apply -Document any bias

Norm Referenced Assessments

-Compare ratings of individual to a normative sample (above or below average) -Outcome: able to evaluate client in relation to group -Use of bell curve

How to use non-standards BEFORE assessment

-Consider potential biases -limit or control potential factors -select assessment relevant to construct being measured

Evaluation requires

-Consideration of occupational performance, client factors, performance skills/patterns, contextual and environmental factors, activity demands -Synthesis of all data obtained -analytical interpretation of data -reflective reasoning

Knowledge needed for effective evaluation

-Diagnosis and conditions -OT constructs -Professional guidelines -Assessments -Resources -Psychometrics

Descriptive Assessment

-Diagnostic -Measures a specific behavioral domain -Tells us whether there is a prob or not -Only given once

How to use non-standards AFTER assessment

-Evaluation findings -Synthesize info -Connect gathered info on theoretical foundation

Observation

-Includes but is not limited to: odors/aromas, temp/tone/color of skin, affect/mood, postures/gestures, speech patterns/eye contact, manner of dress/grooming, response to situation

Evaluation Provides us with?

-Info about pts needs and problems -Info about uniqueness of individual including contextual and environmental aspects -baseline for treatment to evaluate progress

Non-Standardized Assessments

-Ipsative Referenced -Dynamic -High internal validity -Highly individualized info -NOT generalized to other people

Static Assessments

-Measure ability (ex: grip strength) -No feedback given to client -No adjustments to instructions -One-way relationship (therapist >> client) -Results may or may not facilitate intervention planning

Dynamic Assessments

-Measure processes (ex: learning process) -Feedback given to client -Adjustments made in gradation -Two way relationship (therapist <<>>client) -Results should directly facilitate intervention planning

Criterion Referenced Tests in OT

-Measure response to intervention over time -School function assessment (SFA) -Klein-bell assessment of daily living skills -Kohlmann Evaluation of daily living skills (KELS) -Arnadottir OT-ADL Neurobehavioral Evaluation (A-ONE) -Performance Assessment of Self-Care skills (PASS)

Observation Examples

-Model of Human Occupation Screening Tool (MOHOST)

How to use non-standards DURING assessment

-Monitor personal biases/environmental influences -Monitor persons autonomic sys, energy level -Congruence of body language and actions

Types of Observation

-Naturalistic (natural environment) -Analogue (Simulated environment) -Participant Monitoring

Standardized Assessments

-Norm referenced -Criterion Referenced -Static -Have clear instructions for administration and scoring -Can be generalized to general population

Occupational Performance Assessment

-Primary method of data collection -requires activity analysis -often includes rating component

Interview

-Process of inquiry -Informal to formal -Open-ended or closed-ended questions -Info factors heavily into clinical reasoning

Questionnaire

-Self-report that can be completed with or without OT present -Provides info from clients perspective

Assessment

-Should address occupational performance and factors that support occupational performance -Selection should involve several components (from assessment critique)

Criterion-Referenced Approach

-Test results are interpreted by content mastery -Determines what they can do/know -NOT compared with others/average -Show presence or absence of characteristic, ability, or skill -all-or-none score (mastered or not mastered) -need to meet cut score/minimum to passed

Evaluative Assessment

-Useful for detecting clinical change -Grades behavior or performance -Pre-post -Most common

Predictive Assessment

-Useful for prognosis or outcomes -Compares performance/behavior against a criterion -Ex: Fall Screen

Correlation co-efficient > r

-Varies from -1.00 to +1.00 -strength determined by how close r -value is to +/- 1.00 -not a percentage (to find percentage square co-efficient)

Why choose non-standardized measure

-You want individualized perspective of a clients performance -Limited resources, easier to administer, inexpensive, takes less time, does not require alot of training

Concurrent Validity

-compare to gold-standard test

Reliability

-degree of consistency between two test administrations (stability) -Evaluator finds same result each time they conduct test

Validity

-degree to which a test accurately measures the specific construct trait, behavior, or performance it was designed to measure -is it measuring what it claims to measure (is it on target)

Content Validity

-descriptive not statistical -items represent a sufficient representative sample of domain/construct being examined -literature-supported

Predictive Validity

-makes predictions about future behaviors -extent to which scores on current test forecast measure of future criterion ex: fall risk screen

Test Re-test reliability

-measure of test score stability on the same version of the test repeated over two occasions -want a "not significant" result

Face Validity

-no statistical measure, w/o statistical proof -based on appearance -subjective, informal

Evaluation

-only OT may perform evaluation -Plan of care must be based on eval -must have face-to-face interaction w/ client during eval

Test Reliability

-reported using correlation statistic -examines consistency of results -test can be reliable but not valid

Screening

-used to determine need for eval -not reimbursable typically -used to generate referral for OT -Short time period (15 min, 1 unit of service) -Must comply with current state laws and regulatory requirements -may be preformed by OT or COTA (in texas)

Error: Logical Error

Behaviors/performance rated more alike or dissimilar than actually are b/c of info bias from another source

Duration Recording

Can be quantitative or qualitative length of time of occurrence Amount of time needed for completion length of time on or off task Latency (time behavior is not observed) Document time

Interview Examples

Canadian Occupational Performance Measure (COPM) Occupational Performance History Interview-II (OPHI-II)

Standardization occurs through

Clarification of procedures for administration and scoring Verification of psychometric properties Development of standard criteria or norms

Item Bias determined by

Classical test theory Item response theory (Rasch)

Occupational Performance Example

Cognitive Performance Test (CPT)

Rate Recording

Combo of event and duration recording Document frequency of occurrence by length of time

Qualitative Data Collection Methods for Observation

Detailed description of performance/behaviors observed

Improved client and program outcomes are caused by

Effective decision making in evaluation phase which leads to a clear intervention plan

Error: Halo Effect

Evaluators general impression of the test taker affects evaluators rating of performance

Error: Contrast Error

Evaluators subjective response to certain characteristic of the test taker influences scoring

Collection methods for observation: Types of Quantitative Methods

Event Recording Duration Recording Rate Recording Time Sampling

Analyzing Assessment

FIRST: evaluate validity SECOND: evaluate reliability

Test Sensitivity

How well does instrument identify all those who possess behavior or characteristic is question minimizes false negatives should be high

Test Specificity

Implies that all those manifesting target behavior or characteristic will be identified Minimizes false positives Vital in diagnostic assessments should be high

Types of Non-Standardized Assessments

Interview Questionnaire Observation Occupational Performance Assessment

Types of Errors in Standardized Assessments

Item Bias Participant Variable (Evaluator and Test taker)

What do we need to know about a specific problem

Its parameters/characteristics (what happens, when, how affects functioning, how much assistance)

Normative Assessments in OT

Mini Mental (MMSE) Sensory integration and praxis test (SIPT) Lowenstein OT cognitive assessment (LOTCA) Assessment of motor and process skills (AMPS)

Evaluator Effects on Testing

Observer presence (create inaccurate results) Observer Expectation/Observer Bias (create inaccurate results, severity/leniency, central tendency)

Questionnaire Examples

Occupational Self Assessment Version 2.2 (OSA) Child Occupational Self Assessment Version 2.1 (COSA)

Item Bias

Occurs when individuals of similar abilities perform differently on test item -Can be b/c of content asked or covered, language used

Error: Ambiguity Error

Raters interpretation of response to a test varies from typical interpretation made by others

Summative

Refers to the conclusions drawn from the evaluation or decisions made after a course of intervention (can occur during intervention)

Formative

Refers to the process of data gathering to identify the potential problem areas

What psychometric properties need to be evaluated for criterion referenced test

Reliability Validity Limitations (sample size, demographics)

What psychometric properties need to be evaluated for normative test

Reliability Validity Sample size and demographics Standard Error of Measurement

Event Recording

Simplest, most common Tally occurrences Document frequency of phenomenon

Types of Test Reliability

Test Re-test (stability) Interrater and intrarater Alternate, parallel Internal consistency

Pragmatic Reasoning

To consider practical constraints and social factors

Conditional Reasoning

To integrate all of the pieces specific to person

Interactive Reasoning

To respond to patients cues

Test Taker Variables

Tolerance for pain or discomfort Fatigue/stamina Energy Motivation Anxiety/stress Depression Paranoia Beliefs and views of testing

Ecological Validity

Two components: Verdicality: how well does it predict real-world performance Verisimilitude: how similar is test to real-world conditions

Alternate or Parallel Form

Two equivalent forms of the same test Good if items can be easily remembered or simulate learning

Construct Validity

Types: Convergent and discriminant -tests "goodness of fit" between theorized construct and resulting data from test

Criterion-Related Validity

Types: concurrent and predictive -compares test to external criterion

Norm-referenced tests can be used to establish

ability or disability in relation to normative sample justify services

Factorial Validity

ability to identify interrelated behaviors, abilities, or functions -factor analysis, multi-trait validity

Error: Proximity Error

an unexpected event immediately preceding an assessment triggers a reaction to test item

Clinical Responsiveness

can detect change when change happens

Reliability Measures:Below .70

caution, inadequate or unacceptable corrleation

Scorer Reliability: Intra-rater

consistency in measurement and scoring by evaluator when two test results from two similar situations are correlated

Scorer Reliability: Inter-rater

degree of agreement between two scores from two raters following observation and rating of same subject -(.85 or higher is favorable)

Discriminant Validity

degree to which two tests being conducted are measuring a DIFFERENT construct one group does better than the other Negative correlation looking at opposites (two different constructs)

Convergent Validity

degree to which two tests being conducted are measuring the SAME construct both groups do well Positive correlation

Scientific Reasoning

formulate provisional ideas (diagnosis, developmental milestones)

Evaluation needs to pinpoint and identify

functional skills and problems of individual, current environment and expected environment, persons cultural practices and personal meaning

Rasch Theory

gives continuum -easiest to hardest -should hug line

Validity Measures: .60-.80

high correlation

Reliability Measures: .90-.99

high correlation, preferred

Correlation co-efficient

indicates the degree of agreement/relationship between two measures

Validity Measures: .20- .40

low correlation

Validity Measures: .40-.60

moderate correlation

Time Sampling

more common in research recording behavior at specific intervals Document frequency related to time intervals

Validity Measures: .00-.20

negligible correlation

Internal Consistency

reflects the degree that test items measure the same construct (should be >.80) Types: split half, kuder richardson, coefficient alpha, rasch analysis "one of these things is not like the other"

Reliability Measures:.80-.89

satisfactory or adequate correlation

Narrative Reasoning

to construct a picture of individuals needs and situation (life story, qualitative)

Validity Measures: .80-1.00

very high correlation

Reliability Measures: .70-.79

weak or minimally acceptable correlation

Correlation

what is common between two measures