Threats to Reliability Validity

Measurement Artifacts

-Discontinuous measurement -Poorly scheduled measurement periods -Insensitive and/or limited measurement scales

Statistical Regression

A change from one testing session to the next that results in a reversion to the mean

Unreliability of the Measures

An error in the measurement procedure (is it a valid index that gives reliable results?)

Instrumentation

Any change in the measuring instrument or in the way the administrators use it

Maturation

Any change of the subject over time (getting older, taller, stronger, healthier/sicker, learning things)

Testing

Any change that could be attributed to exposure to the test/measure multiple times

History

Any other events occurring at the same time of the experiment that may have influenced the results or account for the data

Excessive Variability in the Data

Any possible outside source of variability

Indirect Measurement

Any time the researcher needs to make inferences about the behavior (as with indirect measurement) there is threat to the measurement validity

Generality Across Settings

Can the results extend to other situations beyond the testing/training setting?

Generality Across Responses/Measures

Can the results extent to behaviors/domains not included in the study?

Generality Across Behavior Change Agents

Can the results extent to other people who might administer the intervention?

Generality Across Time

Can the results extent to other times of day, or beyond testing/training?

Generality Across Subjects

Can the results extent to subjects who may be different from those in the study?

A behavior analyst asks a classroom teacher to measure the duration of attending of a specific student during a morning circle time. Attending is defined as the student sitting still, facing the teacher, and making eye contact with the teacher. While she is running the circle, the teacher is asked to start the stop watch when the student attends and stop it when-ever the student ceases attending. The teacher is unable to measure and run the morning circle at the same time.

Difficult to use Measurement System

A team is collecting baseline data on a child and a specific target. At the same time, they have been discussing possible interventions for this target. One of the ABA Therapists begins implementing the proposed intervention before the baseline period has ended without reporting this.

Diffusion of Treatment

Insufficient Data

Do you have enough data?

A Case Manager designs an intervention for the target behavior - tantrums. The Case Manager implements the intervention and experiences certain results during his sessions, but notices that the data is much higher with other ABA Therapists, despite their correct implementation of the procedure.

Generality Across Behavior Change Agents

An intervention known as matched stimulation has been used for decreasing motor stereotypy, but has not been used or studied in the research to treat vocal stereotypy.

Generality Across Responses

A BCBA designs an intervention to use in structured speech therapy sessions which results in a child sitting still for long periods of time. The BCBA tells the parent to try the same intervention in the less structured home setting, assuming it will be just as successful there.

Generality Across Settings

Research has suggested that a certain intervention is effective for individuals with severe autism. The intervention has not been used studied for individuals with other types of disabilities or with less severe autism.

Generality Across Subjects

A BCBA uses a punishment procedure to decrease swearing. The intervention has an effect on the behavior & during ABA sessions the behavior decreases. After sessions, the behavior increases and remains at a high level during the period of time following an ABA session.

Generality Across Time

In a classroom, a teacher assistant is measuring the duration of sleep the student demonstrates (teachers are concerned that the student is falling asleep often in class). After a baseline period, the student is prescribed a sleep medication to improve sleep. During the first month of use of this medication, the student and his family move to a new home

History

Inadequate Observer Training

If observers are not sufficiently trained, consistency of data suffers. Observers should be selected carefully, trained sufficiently, and provided with ongoing training.

A behavior analyst designs a data sheet and sends a copy to the ABA therapists via e-mail. She tells them to "look it over and ask her if they have any questions". Neither therapist has extensive experience in measurement.

Inadequate Observer Training

Diffusion of Treatment

Independent variable gets administered at the wrong time

A BCBA designs a measurement system where the therapist records whether a child followed directions - yes or no - at all during a session. This gives a very imprecise measure of the target behavior.

Insensitive Measurement Scale

A BCBA designs a measurement system to count frequency of aggressive behavior. After a month, the BCBA revises the operational definition of aggression because he feels he originally missed several topographies of aggression the student exhibits.

Instrumentation

Mixed Data Patterns

Is the pattern of behavior change variable from one intervention phase to the next?

A child is receiving speech therapy sessions over the course of a year. The team is measuring the student's accuracy with labeling. Some measures recorded the number of items the child can measure at the beginning of services, and again a year later, before a new service plan is developed. The child has changed in many ways over the year; some of these ways have not been targeted in sessions, and some have.

Maturation

A behavior analyst is interested in knowing how long a student can attend to a reading assignment without requiring a prompt to remain on task. She designs a measurement system that requires the ABA Therapist to record how many times a day the student initiates reading (frequency).

Measuring the Wrong Dimension of the target behavior

Jeffrey receives ABA Therapy, Speech Therapy, and attends a day school for children with disabilities. His parent is trying to assess his overall change in use of expressive language, and attribute it to a specific therapy (to determine if all are beneficial or necessary).

Multiple Treatment Interference

An ABA Therapist has told her team members (including the behavior analyst) that he believes John socially interacts more when girls are present versus boys. He has been quite opinionated that John should sit at a table in the classroom with only boys (to increase attending to task) and at a table with girls at lunch (to facilitate social interaction). The BCBA designs a measurement system where the therapist notes the gender of peers present when John socially interacts.

Observer Expectation

A BCBA oversees home services provided twice per week to a student with autism. Once per week the BCBA observes the sessions. During this session, she evaluates how well the ABA provider measures. During the other session, the provider is not observed. The provider is nervous about these observations and a bit more attentive to detail when being observed by the BCBA.

Observer Reactivity

Unintended Influences on Observers

Observer expectations and observer reactivity can impact the consistency and accuracy of data.

A home-based ABA provider wants to measure how often a student interacts verbally during meals. She designs a measurement system that requires the ABA Therapist or a parent to record the number of unprompted verbal statements a child makes during a meal. She has the therapist chose either lunch or dinner to record data during. The meals last roughly the same amount of time, but not exactly. At lunch, the child's nanny, who is very talkative, is present with the child and his siblings. She often tells the children stories while they eat. At dinner, both of the child's parents are present. The child typically has not spent time with either parent before dinner, as both work during the day.

Poorly Scheduled Measurement Periods

A student scored very low on the measurement final examination. They retook the exam, and the second time, scored higher, or closer to the class average.

Regression to the Mean

Reactive Assessment

Subjects may be influenced by their awareness that their behavior is being assessed

Reactive Experimental Arrangements

Subjects may be influenced by their awareness that they are involved in a study

A woman has decided to use self-monitoring to measure her smoking behavior. She begins to record the # of cigarettes she smokes each day. She begins to think about the recording, and wonders if it is making her less likely to smoke throughout the day.

Testing

Measuring the Wrong Dimension of the Target Behavior

The data obtained may not be relevant to the question at hand.

Using a student completed course evaluation to determine the quality of a graduate class results in this threat to validity.

Use of Indirect Measurement

A behavior analyst wants to measure the # of times a student initiates a social interaction. For convenience, she has staff record the # (frequency) of social interactions in a 5-minute time sample. The 5-minute sample is taken from different points in the student's school day - sometimes first thing upon arrival, sometimes during a student's lunch break, sometimes at the end of a teaching session.

Using Discontinuous Measurement

Attention and Contact Accorded to the Client

Was the effect due to increased attention during intervention and/or decreased attention during baseline?

Special Stimulus Conditions, Setting, and Contexts

Was the effect due to intervention being administered by a "special" person or in a "special" place

Trends in the Data

Was there a trend in baseline before the intervention was in place? Or did one begin later after the intervention was implemented?

Multiple-treatment Interference

When a subject is exposed to more than one treatment - results may only extent to others who have had the same combination of treatments

Poorly Designed Measurement System

When measurement systems are unnecessarily difficult to use, observers may have a difficult time consistently detecting behavior.