Midterm

Interobserver agreement

-A procedure for enhancing the believability of data that involves comparing simultaneous but independent observations from two or more observers. -Provides no information about accuracy or reliability. -most common approach to evaluating mmt procedures and data -2nd observer doesn't provide true values, they're just being used to evaluate the primary observer's performance -perfect agreement =/= accurate and =/= that both observers detected the same Rs (both could be wrong) -encourage investigator to accept the data as good enough

verification

-DV changes in predictable direction as the IS is systematically applied -replication is needed to confirm this

attrition

-EV -loss of subjects during the course of a study -stress the need of the individual, during the informed consent process, to commit -participants may die, require surgery, move out -know with whom you're working and whether attrition is likely to happen

maturation

-EV -natural development of an individual over time -limit length of study -obtain as much info as possible on individual's behavior to account for maturation -greater threat for younger participants

history

-EV -the passage of time and unforeseen and foreseen events arise -change in season, family issues, etc -obtain as much info as possible on these changes -seek to control foreseen events -alter presentation of IV so that it's clearly linked to DV

accuracy and reliability

-If a set of observations are determined to be completely accurate, they are also completely reliable. -if the data are fully accurate, there is no reason to even raise the issue of reliability. -reliability is only an issue if there is some error. ---Only then can we ask whether the error is small or large, high or low, or consistent or variable -if data are reliable, they're not necessarily accurate

good quality data

-If low quality mmt system - data won't be any good -If data isn't of good quality then interpretations will not be consistent with what actually happened. -assess quality of mmt system throughout entire study

enhance believability

-Interobserver agreement - most common approach to evaluating mmt procedures and data -other corroborative evidence ---any evidence that may persuade others that the data are good enough.

calibration

-Set up specialized mmt that are designed to eliminate error and compare those measures to your mmt values -assess accuracy and reliability. -use a different device than your mmt system that's been specially designed to eliminate error -calibrate device before and after each experimental session. -you can't calibrate humans, so you can't measure accuracy and reliability of humans -if accuracy and reliability are unacceptable there are 2 choices: 1. abandon experiment, or 2. ID and fix problems w/ mmt procedures (calibration) -should be done routinely

accuracy

-The extent to which observed values approximate to the events that actually occurred. Two requirements for evaluating _______ 1. data must represent real physical events -data must not be the only evidence that what is being measured actually exists 2. It must be possible to measure these events directly

Validity

-The extent to which observed values represent the events they are supposed to represent and that will be the focus of interpretation. -direct measurements are valid -problems with validity arise when indirect measurement is used -is a questionnaire a proper measure of the DV? -address issues by collecting direct measures and comparing them to the indirect measures.

Believability

-The extent to which the investigator can, in the absence of direct evidence, convince others to believe that the data are good enough for interpretation. Does not involve direct evidence about the relationship between data and the events they are intended to represent -data may be worthless, may not represent what actually happened. -may not be feasible to assess relationship between accuracy, reliability, and validity

Reliability

-The stability of the relationship between observed values and the events that actually occurred. -the nature of the error in accuracy -a measurement procedure is reliable if it yields the same result every time it is used to measure the same events -not the same as reliability in social science bc we don't use psychometric test or instrument administered by researchers or clinicians as a way of making statements about general features of behavior or about mental qualities.

ethical and human treatment for kids

-What is a child? -Is a child free to make choices? -Should a child be allowed to make choices concerning interventions and treatments? -Does a child behave in a manner that is observable and predictable in ac- cordance with principles of applied behavior analysis? -Can a child's behavior be changed by external forces? -Can an educator modify a child's behavior? -Can another child or parent modify a child's behavior? -Who shall determine which and whose behavior is to be modified? -Which interventions are to be applied in school or in other settings? -Who will approve the use of these interventions and monitor their ethical use? -What are the outcomes sought?

ethical and human treatment

-Who shall decide who will manage behavior? -Who shall decide whose behavior is to be changed? -Who will guarantee that the behavior manager behaves ethically? -What type of interventions will be used? -Who will determine if these interventions are ethical? -What are the outcomes sought?

extraneous variables

-anything that may affect the demonstration of a functional relationship between the IV and the DV -Cooper, Heron, and Heward (2007) noted that the strength of an experimental design is evident to the extent that it reduces or eliminates the influence of confounding variables while still allowing the researcher to investigate the research questions. -ex: maturation, multiple interventions, extent to which IV is applied consistently and as planned, history, attrition

guidelines for ethics

-by Walker and Shea (1991) -Explore alternative interventions before selecting aversives -Consider potential side effects and injury that may occur as a consequence of any intervention -Determine whether the individual understands the treatment program -Anyone involved in applying an intervention should be trained and comfortable with the procedures -Empirical evidence should be available indicating that the intervention is effective -Any formal plans should be consistent with the planned treatment and should be agreed to by the principals involved in those plans -The planned intervention should be carefully monitored, its results should be documented, and it should be regularly evaluated. -Informed consent should be obtained and include information about the nature of the program, benefits, risks, expected outcomes, and possible alternatives to the planned treatment -The principle of normalization should be applied. The procedure used should be fair and appropriate to the degree of con- cern regarding the behavior targeted for change (Allen, 1969; Sabatino, 1983). In other words, are due process safeguards in place and will the outcomes result in an improved life for the individual whose behavior is changing (Walker & Shea, 1991)? -The dignity and fundamental human worth of the individual should be protected -Committee review of all procedures should occur. Peer review of procedures is also recommended -The principle of the least restrictive environment should be applied.

intervention fidelity

-concerned with a trained observer rating the implementation of the intervention to ensure that it is being carried out correctly -assess implementation of key components of intervention -adequate time allocation for implementation of procedures -coverage of a specific amount of material when appropriate -use 2 observers to rate, compare ratings and verify intervention fidelity -the more people involved the greater the possibility of a lack of intervention fidelity -key component to internal validity (along with IA)

validity in social sciences

-criterion validity - refers to comparing the procedure under examination to an existing instrument that is already accepted as an adequate indicator of the characteristics of interest -content validity - the validity of a measurement procedure is assessed by inspecting its content, rather than by using an empirical method. The usefulness of this technique clearly depends on who is doing the assessment -Construct validity - how well the test measures the underlying construct it is intended to measure. This approach is also quite subjective because it involves the hypothetical nature of the underlying construct.

social validity

-degree to which other people think that the targeted changes in behavior are important and that the methods used to encourage behavior change are acceptable. -the magnitude of change in the dependent variable be socially important; -the implementation of the independent variable is cost-effective, practical, and humane; -those implementing the intervention are typical change agents (e.g., psychologist, teacher, social worker); -changes in the dependent variable are maintained over time; -environmental and social contexts of the study are also typical

internal validity

-degree to which the researcher has adequately controlled the IV, DV, and EVs so that changes in DV are directly attributable to the IV -diminished if some other variable is in effect at same time as IV

external validity

-degree to which the researcher may have confidence that she/he or other researchers will obtain the same or similar results if they use the same or very similar experimental procedures with other individuals, with other target behaviors, or in other settings. -replication of procedures to see if same or similar effects are achieved. -direct replication -systematic replication -educational significance - results should merit conclusions that the interventions used also translated into real world significance (social sig.)

standards for assessment

-depends on certain strengths and weaknesses in a study -depends on uses to which a study's conclusions may be put -what impact will the outcome have on how data is interpreted? -fix problems with procedures: revise definitions, modify mmt procedures --if new data can't be interpreted in same way as old data, discard old data

Exact agreement

-dividing the observation period into intervals in which two observers record the actual number of responses. In order to obtain percent agreement, only intervals in which the two observers agreed on the exact count are considered agreements. -better if using short intervals -(total agreements/total # of ints.) x100 = % agreement

direct replication

-duplicate procedures as precisely as possible -intrasubject replication - same participants used in subsequent study -interobserver replication - different participants used in subsequent study ---more common bc it shows generalization across participants.

when to assess data

-evaluate data as soon as the project begins --discover problems early and minimize amount of data that will have to be discarded --problems w/ procedures can occur at any point --evaluate data under all circumstances in a study -frequency of evaluations depends on complexity of procedures --arbitrary standard = 1/3 of all data --more frequent if observers need to also record the status of tx conditions or bxs of indivs other than the participants --the better the assessment outcomes, the less often they need to be done

interobserver agreement

-evidence that DV is accurate -need to have a good definition for the bx -observer drift can occur -steps: ID observers, train them (ID bx, operationally define the bx, practice), select a mmt system, observers compare their results, calculate IA ---comparison process serves to more precisely define the target behavior, to refine the measurement procedure, and to provide confidence that a reliable system is in place. -if possible, observers should be blind to the purpose of the study and should not calculate IA. -key component to internal validity (along with IF)

prediction verification replication

-if _______________ can be demonstrated, the functional relation bt the IV and DV is evident. -rule out the EVs and systematic bias -key elements in demonstrating a functional relation

sequence of assessment efforts

-if data can be shown to be completely accurate, there is not question about their suitability for research -there's no clear standard in judging the balance bt accuracy and reliability -if degree of error is acceptable, turn to stability of error (is the error consistent from mmt to mmt?) -determine the characteristics of the error (how large, how consistent) -somewhat inaccurate but fairly reliable = usable -unacceptably inaccurate but highly reliable = can correct data by adding/subtracting the error -moderately inaccurate and unreliable = not usable

assessing validity

-if target bx can be directly measured, the data are valid by default bc they represent exactly what investigators are drawing conclusions about. -2 approaches to assess validity of indirect measures: 1. arrange for direct mmt of the target bx on a periodic basis ----use observers/equipment when you can't be there to measure ----measure response products 2. collect corroborative evidence consistent with the assumption of validity - more ambiguous than the first option

prediction

-if there's no effect attributable to the IV, the DVs data path will remain unchanged. -verification of effects of IV on DV occurs when a data path changes in conjunction with a phase change

reactivity

-individual altering their bx as a response to being observed -diminishes with more observations -reduce with practice observations, but presence of observers before introducing IV may have confounding effects -extend baseline until there is stability and a reasonable assumption that the performance was unaffected by the observer -video/audio record, two way mirror, etc.

reliability and validity

-interobserver agreement -internal validity -intervention fidelity -external validity

IOA

-no info on accuracy and reliability, it gives info on believability. IOA is not calibration. -used when you can't calibrate (humans) -makes data more believable calculations -Total count calculation - at end of session you take the total # of obs for each observer. Divide the smaller by the larger and multiply by 100. -Mean count IOA - divide the experimental session into equal segments of time, then calculate proportion of agreement in each of those segments. ((Int. 1 IOA + Int. 2 IOA + Int.3 IOA.....)/N intervals) X100. -Exact count - ((number of ints w/100% agreement)/ # of intervals) x100.

assessing accuracy

-observed values are evaluated by comparing them to true values -determine procedures to obtain true values ---must be different from procedures used to obtain observed values ---procedures must have step to avoid/remove error ---record p's bx with video, permanent product -measures of individual responses are often summarized into session totals and compared to true values ---you can only draw conclusions about accuracy of session totals -can video record while observer collects data and have a different, highly skilled observer score the video tape. compare the two (calculate a correlation coefficient)

observer drift

-observer uses their own personal judgement to decide whether a bx occurred. -change in interpretation of when the target bx occurred or not from the original definition -occurs after a number of observations and if IA has not been regularly reviewed -prevent with regular communication and training

treatment drift

-refers to individuals involved in administering the independent variable producing personal modifications (consciously or unconsciously) that may influence the impact of the independent variable on the dependent variable -use same precautions as observer drift

accuracy

-refers to the extent to which the observed value approaches the true value (how much they match) -the amount of error bt the observed value and true value

empirical validity

-refers to the measurements that actually demonstrate that the proposed behavioral changes will indeed positively affect the individual's life -charting of individual behavioral responses may leave unanswered questions bc such measures do not answer whether clinically or educationally significant changes have occurred; if good or bad side effects have occurred; or whether the interventions used were appropriate and humane and carried out in accordance with philosophical and legal assumptions regarding the rights of individuals

replication

-repeating of the observed predictions and verifications within the same study -return to baseline (ABAB) -the repititions, the less likely that changes in DV are due to EVs. -strengthens robustness of IV

reporting assessment results

-report evidence about the quality of data in a way that allows others to make their own evaluations -methods section should describe data assessment procedures -how were true values obtained? -how were observers trained if IA was used. -describe frequency and distribution of assessment procedures -use tables and visual aides.

ethics

-single subject research emphasizes change with individual participants and the value placed on those changes by them and their SO's. -common to obtain data that are not numerical ---interviews, observations with SO's ---compilation and presentation of data from multiple sources to verify social validity -social and empirical validity

validity

-the extent to which what you're measuring represents what you intended to measure (doesn't apply to BA) -in BA if we use a survey we wouldn't say it's anything except verbal bx. We never claim to directly measure something indirectly. All our measurements are direct. There's no such thing as an indirect measure in BA, so that def of validity doesn't apply to BA. -doesn't matter in BA bc we don't indirectly measure things. -is what we're calling what we're measuring consistent with the concept of what people would agree is that. (won't use it much in BA).

Believability

-the extent to which you're confident what's being reported is consistent with what actually happened.

reliability

-the extent to which your accuracy remains consistent over repeated measures. -high reliability - error will remain consistent -more important than accuracy bc the fluctuations of the DV are what's most important. If reliability is low the fluctuations are not representative of the DV, they're representative of changes in your mmt system.

systematic bias

-the researcher has introduced some element to the experimental conditions that is likely to influence changes in the dependent variable regardless of the manipulation of the independent variable -involve more people in reviewing the procedures to avoid this

observed values

-the value that's return by your mmt system -Values resulting from observation and recording procedures used to collect the data for a study -numbers representing some amount of a quantitative dimension -data of a study

goal of scientific mmt

-to arrive at the best possible estimate of the true value of some event that has occurred. --you'll never be able to record the actual true value, there will always be error --admit possibility that data include some error --encourages researchers to obtain the best estimate possible.

Total agreement

-used for count, duration, and latency (QDs) -summing the total count for each of two observers, dividing the smaller total by the larger total, and multiplying the result by 100 to arrive at the percent agreement. -(smaller total/larger total) x100 = % agreement

Interval agreement

-used for interval recording/ time sampling -Each interval scored by two observers is counted as an agreement, and each interval that is scored by neither observer is also called an agreement. Intervals for which only one observer scored the behavior are counted as disagreements. -(total agreements/total # of ints.) x100 = % agreement -if responding occurs very frequently/infrequently the number of agreements could be misleading (% agreement would appear very high

oc/nonoc agreement

-used for interval recording/ time sampling -solution to problem with interval agreement -conservative approach that involves calculating and reporting agreement separately for both occurrences (scored intervals) and nonoccurrences (unscored intervals). -prevent inflation of scores

systematic replication

-varying the conditions from an earlier study, but obtaining similar results -enhances generality of the procedures -if different results are obtained, you'll be unable to discern which variation caused the results

single subject designs

-versatile and flexible -allow for the use of a variety of data collection and presentation techniques -may involve an individual or a number of persons -focus on socially valid changes -incorporate a wide variety of interventions and outcome measures -are applicable in various educational settings -have been used with students across all ages and with many types of strengths and challenges -provide for both internal and external validity -are relatively easy to understand.

true values

-what actually happened -Values resulting from special observation and recording procedures that are somewhat different from those used to collect the data being evaluated and that involve special efforts to minimize error -take special precautions to ensure that sources of error have been avoided or minimized in some way -precautions taken depend on the target bx and how it's being observed -no guarantee that data are error free

multiple treatment interference

-when more than one IV is used -ensure that IV is carried out consistently, otherwise treatment drift may occur -intervention fidelity

assessing reliability

1. Obtain true values and compare with observed values -byproduct of assessing accuracy -if consistency of error bt observed and true values is high, data is acceptably reliable. -same methods for assessing accuracy 2. Present observer with same sample multiple times -present observer with video of the bx 2+ times to evaluate the consistency of observational judgements -p's bx is unchanging from one viewing to another

procedures for determining IA

1. Select and train primary and secondary observers 2. Set up independent observation procedure -each observer must not be able to tell when the other is detecting and recording a response 3. Select agreement formula and calculate agreement -depends on what aspect of the bx is observed

IA formulas

1. Total agreement -used for count, duration, and latency (QDs) -summing the total count for each of two observers, dividing the smaller total by the larger total, and multiplying the result by 100 to arrive at the percent agreement. -(smaller total/larger total) x100 = % agreement 2. Exact agreement -dividing the observation period into intervals in which two observers record the actual number of responses. In order to obtain percent agreement, only intervals in which the two observers agreed on the exact count are considered agreements. -better if using short intervals -(total agreements/total # of ints.) x100 = % agreement 3. Interval agreement -used for interval recording/ time sampling -Each interval scored by two observers is counted as an agreement, and each interval that is scored by neither observer is also called an agreement. Intervals for which only one observer scored the behavior are counted as disagreements. -(total agreements/total # of ints.) x100 = % agreement -if responding occurs very frequently/infrequently the number of agreements could be misleading (% agreement would appear very high 4. occurrence/nonoccurence agreement -used for interval recording/ time sampling -solution to problem with interval agreement -conservative approach that involves calculating and reporting agreement separately for both occurrences (scored intervals) and nonoccurrences (unscored intervals). -prevent inflation of scores

assessing the quality of mmt system

3 parameters for assessing the quality of mmt system 1. accuracy - refers to the extent to which the observed value approaches the true value (how much they match) -the amount of error bt the observed value and true value 2. reliability - the extent to which your accuracy remains consistent over repeated measures. -high reliability - error will remain consistent -more important than accuracy bc the fluctuations of the DV are what's most important. If reliability is low the fluctuations are not representative of the DV, they're representative of changes in your mmt system. 3. validity - the extent to which what you're measuring represents what you intended to measure (doesn't apply to BA) -in BA if we use a survey we wouldn't say it's anything except verbal bx. We never claim to directly measure something indirectly. All our measurements are direct. There's no such thing as an indirect measure in BA, so that def of validity doesn't apply to BA. -doesn't matter in BA bc we don't indirectly measure things. -is what we're calling what we're measuring consistent with the concept of what people would agree is that. (won't use it much in BA).