Chapter 5
total duration IOA
A relevant index of IOA for total duration measurement; computed by dividing the shorter of the two durations reported by the observers by the longer duration and multiplying by 100
Reporting accuracy assessments
Describing procedures used to assess the accuracy of the data, researchers should report the number and percentage of measures that were checked for accuracy, the degree of accuracy found, the extent of measurement error detected, and whether those measurement errors were corrected in the data.
Accuracy assessment procedures
Determining the accuracy of measurement is a straightforward process of calculating the correspondence of each measure, or datum, assess to its true value.
Measurement artifact
Discontinuous measurement Poorly scheduled measurement periods Using insensitive or limiting measurement scales
continuous measurement
Measurement conducted in a manner such that all instances of the response class (es) of interest are detected during the observation period.
Continuous measurement
Measurement conducted in a manner such that all instances of the response class(es) in interest are detected during the observation period.
discontinuous measurement
Measurement conducted in a manner such that some instances of the response class(es) on interest may not be detected.
Observer reactivity
Measurement error resulting from an observer's awareness that others are evaluating the data he/she reports.
Assessing the accuracy of measurement
Measurement is accurate when the observed values (i.e., the numbers obtained by measuring an event) match the true values of the event. First, it is important to determine early in an analysis whether the data are good enough to serve as the basis for making experimental or treatment decisions. The first person that the researcher or practitioner must try to convince that the data are accurate is herself. Second, accuracy assessments enable the discovery and correction of specific instances of measurement error.
Assessing the reliability of measurement
Measurement is reliable when it yields the same values across repeated measures of the same event. Reliability is established when the same observer measures the same data set repeatedly from archived response products such as audiovisual products and other forms of permanent products. The more frequently a consistent pattern of observation is produced, the more reliable the measurement. n
measurement bias
Nonrandom measurement error; a form of inaccurate measurement in which the data consistently overestimate or underestimate the true value of an event.
Indirect measurement
"What is actually measured is in some way different from" the target behavior of interest.
Total count IOA
A percentage of agreement between the total number of responses recorded by two observers and is calculated by dividing the smaller of the counts by the larger count and multiplying by 100.
Direct measurement
"the phenomenon that is the focus of the experiment is exactly the same as the phenomenon being measured"
true value
A measure accepted as a quantitative description of the true state of some dimensional quantity of an event as it exists in nature. Obtaining it requires "special or extraordinary precautions to ensure that all possible sources of errors have been avoided or removed.
True value
A measure obtained by procedures that are independent of and different from the procedures that produced the data being evaluated and for which the researcher has taken "special or extraordinary precautions to insure that all possible sources of error have been avoided or removed".
observed value
A measure produced by an observation and measurement system. It serve as the data that the researcher and other will interpret to from conclusions about an investigation.
Naive observer
A trained observer who is unaware of the study's purpose and/or experimental conditions in effect during a given phase or observation period.
trial-by-trial IOA
An IOA index for discrete trail data based on comparing the observers' counts (0 or 1) on a trial-by-trail, or item-by-item, basis; yields a more conservative and meaningful index of IOA for discrete trial data tan does total count IOA
mean duration-per-occurence IOA
An IOA index for duration per occurrence data; also a more conservative and usually more meaningful assessment of IOA for total duration data calculated for a given session or measurement period by computing the average percentage of agreement of the durations reported by two observers for each occurrence of the target behavior.
interval-by-interval IOA
An index of the agreement between observers for data obtained by interval recording or time sampling measurement; calculated for a given session or measurement period by comparing the two observer's recordings of the occurrence or nonoccurrence of the behavior in each observation interval and dividing the number of intervals of agreement by the total number of intervals ad multiplying by 100. Also called the point-by-point or total interval IOA.
unscored-interval IOA
An inter observer agreement index based only on the intervals i which either observer recorded the nonoccurence of the behavior; calculated by dividing the number of intervals in which the two observers agreed that the behavior did not occur by number of intervals in which either or both observers recorded the nonoccurence of the behavior and multiplying by 100. Unscored-interval IOA is recommended as a measure of agreement for behavior the occur at hight rates because it ignores the intervals in which agreement by chance is highly likely.
scored interval IOA
An interobserver agreement index based only on the intervals in which either observer recorded the occurrence of the behavior; calculated by dividing the number of intervals in which the two observers agreed that the behavior occurred by the number of intervals in which either or both observers recoded the occurrence of the behavior and multiplying by 100. It is recommended as a measure of agreement for behaviors that occur at low rates because it ignores the intervals in which agreement by chance is highly likely.
naive observer
An observer who is unaware of the study's purpose and /or the experimental conditions in effect during a given phase or observation period. Data obtained by this observer are less likely to be influences by observers' expectations.
Discontinuous measurement
Any form of measurement in which some instances of the response class (es) of interest may not be detected. No matter how accurate and reliable - may yield data that are an artifact.
calibration
Any procedure used to evaluate the accuracy of a measurement system and, when sources of error are found, to use that information to correct or improve the measurement system.
observer drift
Any unintended change in the way an observer uses a measurement system over the course of an investigation that results in measurement error; often entails a shift in the observer's interpretation of the original definitions of the target behavior subsequent to being trained.
Scheduling observation & recording periods
Because you should always be mindful of the validity (i.e. method measured what it is supposed to measure) of your data, you have to be very careful in planing out your observation & recording periods.
Inadequate Observer Training
Careful attention must be paid to the selection and training of observers. Explicit and systematic training of observers is essential for the collection of trustworthy data.
Valid Measurement
In applied behavior analysis requires three equally important elements: 1. Measuring directly a socially significant target behavior 2. Measuring a dimension (e.g. rate, duration) of the target behavior relevant to the question or concern about the behavior 3. Ensuring that the data are representative of the behavior's occurrence under conditions and during times that are most relevant to the question or concern about the behavior. When any of these elements are suspect or lacking - no matter how technically proficient (i.e. accurate and reliable) was the measurement that produced the data- the validity of the resultant data are compromised, perhaps to the point to being meaninglessness.
observer reactivity
Influence on the data reported by an observer that results from the observer's awareness that others are evaluating the data he reports.
Observers must use the same measurement system
Interobserver agreement assessments conducted for any of the four previously stated reasons require observers to use the same definitions and codes, and measurement devices.
Total Duration IOA
Is commuted by dividing the shorter of two durations reported by the observers by the longer duration and multiplying by 100.
Mean Duration-per-Occurrence IOA
It is calculated for duration per occurrence data, and it is a more conservative and usually more meaningful assessment of IOA for total duration data.
Validity
It yields data that are directly relevant to the phenomenon measured and to the reason (s) for measuring it.
Training observers to an Objective standard of Competency
Observer trainees should meet a specified criterion for recording before conducting observation in applied settings. During training, observers should practice recording numerous examples and non examples of the target behavior(s) and receive a critique and performance feedback.
Observer drift
Observers sometimes later, often unknowingly, the way they apply a measurement system. The unintended changes in the way data are collected may produce measurement error. It usually entails a shift in the observer's interpretation of the definition of the target behavior from that used in training. It occurs when observers expand or compress the original definition of the target behavior.
Benefits and Uses of IOA
Obtaining and reporting interobserver agreement serves four distinct purposes, 1. A certain level of IOA can be used as a basis for determining the competence of new observers. 2. Systematic assessment of IOA over the course of a study can detect observer drift 3. Knowing that two or more observers consistently obtained similar data increases confidence that the definition of the target behavior was clear and unambiguous and the measurement code and system not too difficult 4. For studies that employ multiple observers as data collectors, consistently high levels of IOA increase confidence that variability hight levels of IOA increase confidence that variability in the data is not a function of which observer (s) happened to be on duty for any given session, and therefore that changes in that data are more likely reflect changes in the behavior.
indirect measurement
Occurs when the behavior that is measured is in some way different from the behavior of interest; considered less valid than direct measurement because inferences about the relation between the data obtained and the actual behavior of interest are required.
direct measurement
Occurs when the behavior that is measured is the same as the behavior that is the focus of the investigation.
Unscored-Interval IOA
Only intervals in which either or both observers recorded the nonoccurence of the target behavior are considered when calculating agreement. An agreement is counted when both observers recorded the nonoccurence of the behavior in the same interval, and each interval in which one observer recorded the nonoccurence of the behavior and the other recorded its occurrence is counted as a disagreement.
Scored-Interval IOA
Only those intervals in which either or both observers recorded the occurrence of the target behavior are used in calculating this agreement. An agreement is counted when both observers recorded that the behavior occurred in the same interval, and each interval in which one observer recorded the occurrence of the behavior and the other recorded its nonoccurence is counted as a disagreement.
Selecting observers carefully
Potential observers should be interviewed to determine past experiences with observation and measurement activities, currency schedule and upcoming commitments, work ethic and motivation, and overall social skills.
Measurement bias
Refers to nonrandom measurement error; that is, error in measurement that is likely to be in one direction. When measurement error is random, it is just as likely to overestimate the true value of an event as it is to underestimate it.
reliability
Refers to the consistency of measurement, specifically, the extent to which repeated measurement of the same event yields the same values.
Interobserver agreement (IOA)
Refers to the degree to which two or more independent observers report the same observed values after measuring the same events.
Accuracy
Refers to the extent to which the observed value, the quantitative label produced by measuring an event, matches the true state, or true value, of the event as it exists in nature. In other words, that it corresponds to the true value of the thing measured.
Trial-by-Trial IOA
The agreement between two observers who measured the occurrence or nonoccurrence of discrete trip behaviors for which the count for each trial, or response opportunity, can only be 0 or 1 can be calculated by comparing the observers' total counts or by comparing their counts on a trail-by-tral basis.
mean count-per-Interval IOA
The average percentage of agreement between the counts reported by two observers in a measurement period comprised of a series of smaller counting times; a more conservative measure if IOA than total count IOA
Threats to measurement accuracy and reliability
The biggest threat to the accuracy and reliability of data in applied behavior analysis is human error. Poorly designed measurement systems, inadequate observer training, and expectations about what the data should link look like.
interobserver agreement (IOA)
The degree to which two or more independent observers report the same observed value after measuring the same event.
Reliability
The extent to which a "measurement procedure yields the same value when brought into repeated contact with the same state of nature". It is consistent measurement.
validity
The extent to which data obtained from measurement are directly relevant to the target behavior of interest and to the reason(s) for measuring it.
accuracy
The extent to which observed values, the data produced by measuring an event, match the true state, or true values, of the event as it exists in nature
Believability
The extent to which the researcher convinces herself and others that the data are trustworthy and deserve interpretation. Measures of interobserver agreement (IOA) are the most often used index for believability in applied behavior analysis.
Mean count-per-Interval IOA
The likelihood that significant agreement between observers' count data means they measured the same events can be increased by 1) dividing the total observation period into a series of smaller counting times 2) having the observers record the number of occurrences of the behavior within each interval 3) Calculating the agreement between the observers' counts within each interval 4) using the agreements per interval as the basis for calculating the IOA for the total observation period
Poorly Scheduled Measurement Periods
The observation schedule should be standardized to provide equal opportunity for the occurrence or nonoccurrence of the behavior across sessions and consistent environmental conditions from one observation session to the next. When neither of these requirements is met, the resultant data may no be representative and be invalid.
Observers must measure the same events
The observers must be able to observe the same subject (s) at precisely the same observation intervals and periods. IOA for data obtained by real-time measurement requires that both observers be in the setting simultaneously.
Exact Count-per-Interval IOA
The percentage of total intervals in which two observers recorded the same count.
Exact count-per-Interval IOA
The percentage of total intervals in which two observers recorded the same count; the most stringent description of IOA for most data sets obtained by event recording.
Interval-by-Interval IOA
The primary observer's record for each interval is matched to the secondary observer's record for the same interval.
total count IOA
The simplest indicator of IOA for event recording data; based on comparing the total count recorded by each observer per measurement period; calculated by dividing the smaller of the two observers' counts by the larger count and multiplying by 100.
Establishing true values
There is only one way to assess the accuracy of a set of measures - by comparing observed values to true values. The comparison is relatively easy; the challenge is often obtaining measures of behavior that can legitimately be considered true values.
Requisites for obtaining valid IOA measures
Two observers (usually two, but may be more) must 1. use the same observation code and measurement system 2. observe and measure the same participant(s) and events, 3. observer and record the behavior independent of any influence from one other.
Poorly designed measurement system
Unnecessarily cumbersome and difficult-to-use measurement system create needless loss of accuracy and reliability. Collecting behavioral data in applied sessions requires attention, keen judgment, and perseverance. Simplifying the measurement system as much as possible minimizes measurement errors.
Indicators of trustworthy measurement
Valid Accurate Reliable
Calibration
When measurement error is consistent in direction and value, the data can be adjusted to compensate for the error.
Threats to measurement validity
When the wrong dimension of the target behavior is measured, or when measurement is conducted in such a way that the data it produces are an artifact of the actual events.