CH 9,10,11 Research Quiz #5
Multiple Baselines
-Tracks more than one dependent variable -Inclusion of more than one participant o Able to show performance improvement more widely o Strengths external validity o Might suggest patterns to explain mixed results of intervention o Improves external validity
Sources of error in administration
-test instrument: low reliability, low validity, lack of normative data, cultural bias -administration: improper training, not following directions precisely, incorrect scoring, incorrect interpretation of data, lack of rapport, imprecise use of stop watch and test materials, developmentally inappropriate test -client: lack of motivation, fatigue, language differences or deficits, motor disability, high anxiety, lack of coordination, poor sustained attention, acute psychotic symptoms, poor understanding -environment: noisy, poor lighting, interruptions, poor seating, too hot or cold, poor air quality
Steps in the Evaluation Process
1. Identify the function to be measured 2. Identify a published instrument to measure a condition or develop a new standardized procedure for evaluation of function 3.Identify the skill level necessary to use a test instrument 4. Identify the possible factors in the environment, tester, subject, or test instrument that can potentially distort the test results 5. Identify the target population for which the test is intended and for which norms have been established 6. Strictly follow the directions and procedures for administering and scoring the test or for modifying the test procedure to enable the client with disabilities to perform at a maximum level 7. Interpret the results based on the following: a. Norm-referenced data based on the general population b. Criterion-referenced data for client performance or competence
Single case experimental design
A type of controlled experimental design (not a case report or case study or case series)
treatment fidelity
Is the intervention provided in exactly the intended manner?
characteristics of a good test in clinical research
The accuracy and precision in evaluating a client's performance or improvement in a clinical research study depend on the quality of the test instrument. Essential Questions in evaluating the purposes and "goodness" of a test: ● What is the population for whom the test is targeted? ● What are the specific purposes of the test? ● What are the areas of function identified in the test? ● What are the methods used to evaluate a client? ○ Medical records ○ Educational Records ○ Clinical Observation of Performance ○ Interviews ○ Objective Tests ○ Survey Questionnaires ○ Self-Reports ○ Reports from Family, Peers, Teacher, or Therapist ○ Biomechanical or Physiological Measurement of Human Factors ● Is there a standardized procedure or manual of instructions in administering test and interpreting results? ● Are there tables of normative data? ● Are there special skills or certifications that are necessary to administer, score, and interpret results of the test? ● Is the scale of measurement used in collection data identifiable? ● Are error factors controlled that can potentially interfere with obtaining reliable test results? ● Are research data reported?
observer drift
maintenance throughout the study of intra-rater reliability
Qualifications to administer a test: Level A
no special qualifications to purchase products
Change in level
○ A _______________of the dependent variable from one phase to the next would be indicated by a bump in the graphed line upward or downward ○ Assuming that up is better in this case, the researcher would want to know two things about this bump ■ First, could it have happened purely by chance and not be due to the independent variable? ■ Second, if the level change upward seems to be the new norm for the participant, is the performance now at a level that is functional, akin to normal, or meeting a criterion for task completion, habit consistency, or maintenance of a routine? ■ In other words, as with group experiments the researchers must appraise both the statistical validity and the clinical or occupational validity of the change ○ Statistical procedures adapted for SCED can help answer the first question ○ Only knowledge of human occupation and of the goals of the participant and family can help answer the second
Alternating treatment design structure
○ A different approach is used in the SCED, called _______________ ○ In this design, two interventions are compared on several occasions (usually at least 6) ○ On each occurrence of measurement, one intervention is applied first and then the other is applied; the resulting performance level of the client is then measured ○ If one intervention is clearly superior to the other, the plotted lines will diverge and remain separate
C-Statistic
○ A less often used alternative analytic method for SCED data is the __________ ○ The ______________estimates trends in a time series ○ It needs a minimum of 8 data points in Phase A ○ First it determines whether there is no trend in the baseline ○ Then, if there was not, it determines whether there is a difference between the baseline (A) and the treatment (B) phases ○ All three of the approaches described can be equally used to test for changes from phase B to a return to baseline or from the second phase A to the second phase B, and so on ○ Most authors explicitly warn that these analyses should only be used to compare adjacent phases in the data stream ■ The most important reason for this warning is that the logic of the null hypothesis only applied between a given phase and the immediately preceding phase of data
ABA structure
○ A stronger causal claim could be made if data were further collected during a return-to-baseline phase, where the client's performance sharply returns to that in the original baseline phase when the intervention is removed ○ For interventions providing a non-permanent effect, one expects the dependent variable to decline when the intervention is stopped, such a s in the provision of assistive technology ○ Alternatively, if the intervention seeks to make a permanent change in the client, then one would expect better the better performance to be maintained in the return to baseline (the second Phase A)
Immediate Interpretation
○ Applying visual and statistical analyses to the data points of an SCED puts the researcher in a position to make justifiable inferences about what the experiment has shown ○ These inferences should only be made while keeping in mind the following possible limitations ■ Did the measuring instrument for the dependent variable have noteworthy limitations? ■ Larger than desirable random measurement error in the dependent variable might manifest itself as larger than expected variability in the plotted data ■ If the 2-SD band method of analysis was used, then the larger variability in phase A would result in a wider SD band ■ This greater width would make it less likely for any improvement to have produced two consecutive Phase B points outside the band ■ This disadvantage is analogous to greater variability in a group experimental design, lessening statistical power ○ According to the scale used for the dependent variable, it might display a floor or ceiling effect ■ A floor effect means that the dependent variable scale cannot measure below a certain performance or behavior level ■ Suppose the researcher is seeking to show statistically that the intervention diminished a negative behavior ● The plotted data from Phase A give rise to a 2-SD band that is below the lowest possible score on the dependent variable ● It would then be impossible for Phase B to produce two consecutive data points outside the SD band ● It will appear that the intervention had no noteworthy affect ■ If the change sought is upward, then a ceiling effect would likewise prevent the researcher from using a 2-SD band method to demonstrate the interventions effectiveness ○ Another interpretive limitation occurs when there are missing data points ■ The participant may be ill one day so not data were collected ■ Should the x-axis be collapsed across that missing day? ■ Whether the researcher compresses the x-axis could influence the outcome of data if a split middle binomial test method, when used with only a few data points in Phase A, may become unstable and this be potentially misleading ■ When there are few points in Phase B (fewer than 9), it is difficult or impossible to reject the null hypothesis ○ Mid phase changes in level, slope, or variability pose additional challenges for interpretation ■ First, what could have caused this change? ■ Is it convenient to think that only the independent variable could cause a change in the dependent variable?, but we know that for matters of human performance, this assumption is far too simple ■ Such mid-phase change also creates the following dilemma: should only the data after the change be used to examine subsequent changes between phases A and B? ● Or should all the phase A points still be averaged together to form a representative 2-SD band or sloped line? ■ There is no mathematical answer to these questions ■ The researcher must make the case logically, with reference to the context of the participant being studied
change in level with a positive or negative slope baseline
○ Assume the slope in the data during baseline phase is upward and the intervention was accompanied by a positive bump to the natural healing, learning or developmental process, without the slope changing ■ That would probably indicate an intervention that was successful ○ If the baseline slope was downward and the intervention gave a decline stalling bump to the process, it could also be considered a positive effect, even if the long term downward trend continues ○ Occasionally an intervention will temporarily have a negative effect on a clients performance but will result in greater benefits in the long run ○ Changes in level, slope, and variability must be interpreted considering the salient factors pertaining to the particular situation studied in the SCED ○ When multiple graphs are available, either because of multiple participants or because of multiple baselines, there are more opportunities to examine and interpret similarities and differences or to point out where the data fulfilled the expectations of the researcher and where they did not ○ Such detailed narratives interpreting client changes in functional performance with respect to level, slope and variability- within a participant as well as across participants- virtually never occur in a group experimental design study, with its comparison of group mean scores
Purpose of CRTs and NRTs
○ Compare an individual's achievement or performance with other individuals of the same age, educational level, socioeconomic status ○ Obtain info regarding the normal population
change in slope
○ During the baseline phase, there could be any of three types of slope in the data points ○ They could show essentially no change in level over time (zero slope), they could slope upwards (assuming still that up is better), or they could be sloping downward ○ Now assume the intervention has begun and data are being collecting during Phase B ■ For each of the slopes of Phase A (-,0,+), there could arise a negative, 0, or positive slope in the subsequent Phase B ■ Consider the importance of each of these combinations might be ○ If the baseline slope is 0 and the intervention slope is also 0, we would tend to conclude that the intervention had no effect ○ If baseline slope is 0 and the intervention was accompanied by and upward slope, then we might conclude that the intervention probably had a positive effect on the participants outcome ○ If the baseline slope was already upward, it would take a sharp ramping up of that slope to persuade us that the intervention had a beneficial effect ○ If the baseline slope were downward, we might consider it a victory for the intervention to have changed that slope to zero ○ Of course, the researcher would be most happy if a negative baseline slope were changed by the intervention into a positive upward slope
two standard deviation band method
○ If the data in the baseline phase are stable (have a 0 slope) and are about six or more in number, then the 2-SD band method may be used ○ The analysis is based on the logic of the null hypothesis, which could declare that any difference in level seen between the baseline phase and the intervention phase data is due solely to chance ○ If the data of phase B show enough difference that it is improbable it would have happened by chance, then we can reject the null hypothesis and claim that the difference seen is probably real ○ To use 2-SD band method, the mean value for the points in the baseline phase is found (add the value of the points together and divide by the number of points) ○ A solid line is drawn across Phase A at this level on the graph, then the SD of phase A data points (not A and B) when calculating the SD ○ The size of the SD is then doubled, and this amount is added to and subtracted from the mean value found earlier ○ Dashed or dotted lines are drawn at these two levels across Phase A forming a band ○ This band is 2 SDs above the mean and 2 SDs below the mean (or actually 4 SDs wide) ○ The dashed lines of the 2-SDs band are now drawn across Phase B ○ If 2 consecutive data points from phase B fall outside the band, then one can reject the null hypothesis of there being no real difference between phases A and B ○ Having 2 consecutive points outside the band is held to be too unlikely for the null hypothesis to be retained ○ If the points outside the band are outside in the desired direction, then one may claim that the phase B performance level is better than the Phase A level
Split-Middle/Binomial Test Method
○ If there appears to be a slope in phase A then using the 2-SD band method is not valid because the slope, left alone as it continues across Phase B, would result in higher data points ○ Even if the conditions in the 2-SD band method for rejecting the null hypothesis were triggered, it would not enable a valid claim that the Phase B performance was higher due to the intervention ○ The phase A slope indicated that the participant was already improving before the intervention began ○ Under this condition, a difference in the level between Phases A and B would not be evidence of an effective intervention ○ In this case, the ___________can be used ■ Here the null hypothesis is used once again, but this time with respect to the existing slope in the data across Phase A ■ If the intervention had no effect, then this slope be expected to continue unchanged across Phase B, except for random variation ■ However, if the Phase B points show enough of a deviation from this expectation, then we can reject the null and claim that a change did take place when the intervention was instituted ○ To find the slope representing the overall slope of Phase A, count the number of data points in the baseline, divide the total number if data points in half, and draw a dashed vertical line between the two adjacent points between the first half of points and the second half of points ■ If the baseline had an odd number of points, then draw the vertical line through the middle point leaving an equal number of points to the left and right of it ○ Within each of these half phases, find the median value of the points in that half phase ○ Plot the median value at the position on the x-axis representing the midpoint of time in this half phase ■ Then do this for the second half of phase A ○ Draw a solid line connecting these two points ○ If necessary, adjust this line upward or downward so that there are an equal number of Phase A points above and below the line (this adjustment is often not necessary) ○ Now extend this solid line into and across Phase B ■ Phase B points will lie on either side of this line ■ If the null hypothesis is true, 50% of the points would appear above and below the line (the intervention made no difference) ■ However, it would also not be unlikely if the points divided 60%/40% or even 70%/30% in cases where there are not too many data points in Phase B ■ Indeed, with only 4-8 data points in Phase B, they would all need to be above the solid line drawn from the baseline phase to reject the null hypothesis ■ If there are 9 points in phase B, then one of them could be below the solid line and could reject the null at an alpha of 0.5 ■ For 12 points in Phase B, two of them could be below the solid line and we could still reject the null hypothesis at alpha of 0.5 ■ These decisions are based on the probability values in a binomial test statistical table ○ If the evidence is present to reject the null hypothesis, then the claim can be made that we are 95% confident that performance in Phase B is better (or worse, if the points went that direction) than phase A- and not from pure chance ○ The strength of any argument that the intervention itself caused this difference would need to built up with supporting evidence ○ For ex: if a return to baseline conditions was accompanied by a decline in performance and then a reapplication of the intervention caused an increase, the researcher could be more confident that the intervention is causing the increases ○ Nonetheless, this evidence does not constitute proof that the intervention was the cause It just makes it more plausible that the invention caused the change
Introduction to the research paper
● The first part of the research paper contains a brief ___________ to the present study, including the propose, the research or guiding question(s), and the significance when examined beside findings from landmark studies ● A pivotal part of this portion of the research paper, article, or manuscript is a clear statement of the hypotheses or guiding questions such that they are (a)well understood, (b) lead the researcher to anticipate the major sections of the paper, and © indicate the direction or argument in which the paper will be written ● Finally this part of the paper should include definitions for any terminology that are uncommon or might not be understood by the reader
The phenomenon of serial dependency (autocorrelation)
○ In many group statistical comparisons, one assumption of the statistical test is that the data points are independent of one another ○ An example of non-independence of data would be if children were motor tests where those waiting their turn could see the children being tested before them ○ The performance of one child could influence the performance of another contaminating the independence of the data ○ In SCED, since the entire data stream is derived from the same individual, it is likely that the assumption of independence of data is violated ○ If it is, there could exist the situation of serial dependency or autocorrelation ○ Ottenbachr and others recommended testing for autocorrelation in any data set in SCED ○ Its presence would not affect the C-statistic but could affect the accuracy of the 2-SD band method or the split-middle/binomial test method ○ Therefore, although it is advisable to use statistical analyses to avoid some of the pitfalls of a pure visual analysis, these analyses have their own limitations. ■ They should always be interpreted in context and with caution
SCED characteristics
○ Inspired by an important research question ○ Justify a claim of causal relationship between intervention and outcome ○ Identify demographics of participants ○ Manipulation of variables/factors to isolate causal relationship ○ Operational definitions of variables ○ Procedural rigor (reliability of measures, control of confounding variables) important ○ Triangulate data collection through measuring multiple variables ○ Participants act as their own control ○ Outcomes are appropriate for statistical analysis ○ Conclusion suggest cause-effect relationship
long term interpretation
○ Once the decision has been made whether the Phase A to Phase B transition is accompanied by improved performance, no change, or a decline in performance, and once parallel decisions have been made for every other phase transition (e.g. B to A, A to B), there remains the need to interpret the meaning of the SCED as a whole ○ In a multi-participant SCED, this task would also involve considering why the intervention seemed to work for some individuals but not for others ○ Having demographic and diagnostic information about the participants to hypothesize about possible causes of a difference can be helpful here
ABC/ABAC Structure
○ Phase pairs AB could be repeated further, but more common us the introduction of a second, different intervention, called C ○ The simplest design with a Phase C would be ABC ○ Here the baseline is followed by a period of intervention B followed by a period of intervention C ○ Performance during B might not show an improvement over that of A, but perhaps with the onset of C does not show improvement ○ It would be better if phase C has been preceded by a return to baseline, or ABAC ○ Stronger will would be the design ABACA, where a final return to baseline tests whether the intervention of C was actually effective ■ That is, what happens when it is removed? ■ Does the client's performance drop back to that of the original baseline? ○ It also is possible to design a study where interventions are applied separately and then together (ABACA, then B+C simultaneously) ○ Each additional comparison made between phases offers the opportunity to make a stronger causal case because coincidences are even more unlikely to have caused the change in performance
statistical analysis
○ Some mathematical assumptions of the statistical procedures are not met; thus, it is prudent to interpret any statistical analysis cautiously, with recognition of where limitations may exist ○ There are 3 basic statistical approaches to the analysis of data in SCEDs: ■ 1. The two standard deviation (2-SD) band method- when there is a zero slope in the baseline phase ■ 2. The split middle/binomial test- when the baseline data do show a slope up or down ■ 3. The C- statistic- showing whether there is a trend in the baseline phase and if not whether the intervention data in the subsequent B phase differs from the baseline data
ABAB Structure
○ Stronger yet would be if data were also measured during a second phase B ○ If performance improvement was observed both times there was an A-to-B transition, then the research could be much more confident that the change was caused by the intervention ○ Hence SCRIBE (single case reporting guideline in behavioral intervention) recommendation for contemporary SCED to have a design at least thorough as ABAB
AB structure
○ The simplest SCED has the structure _____ ○ That is a baseline (A) phase of a certain length of time is followed by an intervention (B) phase, also of a certain length of time ○ Each phase involves multiple measurement occasions to provide confidence that the researcher has an accurate estimate if the client's actual performance level during this phase ○ Typically data is measured three to eight times per phase, each time on a different day ○ A minimum of 6 is recommended so that any one data point does not overly influence the findings ○ Whenever a marked difference in performance is apparent after the intervention is started (the transition to Phase B), then the researcher could formulate the claim that the intervention caused the improved performance ○ However, it is possible that something else may have happened just at the phase change that was the actual cause of the improvement ○ This actual cause may be another intervention being started by a different practitioner, a developmental milestone being naturally reached by the client, or an event in the living situation of the client
operationalization of independent variables
○ To replicate or generalize an SCED, readers must be informed about the exact nature of the invention that was delivered to the participant ○ First, to ensure that the intervention was delivered as intended to the participant, procedural reliability is monitored ○ Typically, the monitoring is done via a checklist of every pertinent detail of the setup and delivery of the therapy, as well as of the surroundings ○ Is furniture exactly in its intended place? ○ Are supplies laid out the same each time a data measurement occurs? ○ Are seating, lighting, temperature and background noise standardized from session to session? ○ Is the intervention provided in exactly the intended manner (also called treatment fidelity)? ○ The demands of fidelity can run the gamut from a precisely specified protocol to a globally outlined procedure such as "intervention was applied according to the professional judgment of the treating therapist" ○ If the intervention is provided under requirements more similar to the latter description, then the representativeness of the occupational therapy practitioner providing the intervention becomes of greater importance ○ Would the average therapist implement such an intervention in approximately the same way? ■ To the extent the answer to this question is "no", the generalizability of the findings to other settings is correspondingly weakened ○ Of course if the SCED protocol includes the provision of more than one type of intervention, then each intervention must be operationalized
change in variability
○ Variability itself is not often a concern in therapy situations, however, it may be the key consideration ○ For ex: suppose a young man or a young woman had a flat affect due to an emotional trauma ■ Little to no variation in facial expression could lead to notable disadvantages in social communication and participation ■ The goal for therapy, and thus the focus of analysis in a corresponding SCED, would be to increase the variability of facial expressions ○ A different situation would be where a person is experiencing marked role dysfunction due to alternating periods of mania and depression ■ Here the therapist, and likewise the SCED, researcher would seek to dampen variability of mood experienced by the person ■ Thus, it may be beneficial for the variability of behavior to be increased in some studies and decreased in others, depending on the condition and the situation ■ In either case, an SCED with its repeated measurements, could be the means for persuasively demonstrating that the intervention probably had the desired effect
operationalization of dependent variables
○ When were measurements taken?, By whom were the measurements made?, Did the research protocol include provisions for controlling observer drift? ■ Observer drift- maintenance throughout the study of intra-rater reliability ○ If more than one data gatherer or rater were involved, did the researcher take steps to establish, measure, and maintain inter-rater reliability? ○ These reliabilities are more important in SCED due to the repeated measurement of the same individuals in the generation of the data ○ If observer drift occurs, then an apparent change in performance displayed by the data might not have been caused by the intervention, but rather could be attributable to the drift (systematic measurement error) instead
Bias
● 1st the writer will want to make sure that no type of _________(gender, sexual orientation, racial, ethnic, disability, age, or historical and interpretive inaccuracies) has been introduced in the paper or study. ● "Part of writing without ________ is recognizing that differences should be mentioned only when relevant. Marital status, sexual orientation, racial and ethnic identity, or the fact that a person has a disability should not be mentioned" (APA, 2011) ● Gender refers to the role, whereas sex is biological ○ Both gender and sexual bias can be avoided by specifically describing participants, rather than using a general term to describe the population ○ Ex: bias occurs when the writer uses the term man to mean human race rather than using less ambiguous term: person, people, individual ○ Use of specific gender words (his, her) or using terms that end in man or men (policemen, chairmen, postman) must be avoided ○ Stereotyping can be avoided by using specific language (boys enjoyed playing Legos while the girls enjoyed reading), using parallel language (men and women rather than men and girls), or by changing the term to a plural (their instead of his/her) ○ Compounds such as his/her or (s)he should be avoid ○ Term sexual orientation is preferred over sexual preference ● ________ involving disability occurs when the writer attributes to a specific group something that has no supporting data- ex: "in general, all individuals with TBI tend to have psychosocial difficulties" ● Ethnic and racial bias can be avoided by using commonly accepted designations such as the census categories, while being sensitive to the preferred designations ○ Ex: persons of Hispanic descent in NM might prefer to be called Hispanic or Spanish, while some in southern California might prefer Mexicans ● Avoid terms that are either dated or derogatory ● Terms used to refer to racial or ethnic groups are proper nouns-should be capitalize ● Asking participants their preference, ethnic, or racial bias can be avoid ● Individuals aged 12 years and younger should be referred to as girls and boys, 13-18 is young men or young women, and those older than 18 are women or men ○ Elderly or senior terms are generally not accepted
Mechanics of writing
● A 3rd reason to revise a manuscript is to make sure that there are no grammatical errors ● Run on sentences and sentence fragments are an indication that either the writer is careless & has not proofread or needs help with writing skills ● Other common errors include ○ (a) failing to use parallel structure for grammatical units ○ (b) having dangling modifiers ○ © splitting infinitives ○ (d) improperly using relative pronouns ○ (e) using the passive tense rather than the active tense ○ (f) placing prepositions at the end of sentences ● A 4th reason for revision is to check for spelling and punctuation errors ● Abbreviations invented by the writer should be kept to a minimum and used only for terms that are long and frequently repeated
Transitional Words, Phrases, Sentences
● A 5th reason for revision is to make sure that transitional statements are present so that the flow of ideas is smooth ● Writing a well-organized, integrated manuscript requires the use of words and phrases that connect ideas and summarize a section of a paper ● In reviewing scientific literature, it is important to integrate the studies under topical areas and to develop a logical sequence- writing should be coherent ● Note that many style guides frown on beginning a sentence with however
People first language
● A second reason for revision is to make sure that the writing is in ________________ ● Because of the Individuals with Disabilities Education Act (1990) all references to disabilities must be written with the individual placed first ● This convention emphasizes the value of the human being and delegates the disability to secondary importance ● The writer must distinguish also between the terms handicap and disability ○ A handicap can be thought of as a disadvantage imposed on an individual by the environment or by another person ○ A disability is something that an individual is unable to do, a lack of body part, or physical or psychosocial impairment ■ Ex: an individual with SCI may not be able to walk; the inability to walk is a disability and can become handicap if there is no w/c access. ■ The presence of a disability does not make the individual handicap
Visual and statistical analysis
● A valuable aspect of SCED is that three types of changes between phases can be detected: change in level, change in slope, and change in variability ● These changes could occur one at a time, two at a time or all three at once ● In traditional group experimental design, the main attention is placed upon changes in level ● Much less often is attention placed in a group design on differences in slope between the groups or on differences in variability between the two groups ● There is nothing that prevents a researcher from building these analyses into a group design ● However, an SCED more readily lends itself to examining both level and slope, and sometimes in addition variability, given the way data are collected and displayed
Role of SCED in the Profession
● As their participant selection was rarely anything other than a convenience sample, one cannot make a statistical projection about the generalizability of their findings to the larger population ● However, their findings do offer other evidence about practice that is of value to therapists ○ First, some SCEDs allow a generalization from one or a few cases to a hypothesis- that is why a certain intervention was effective for a person or why there was a difference in response among different individuals ● PPT from class: ○ Allow a generalization from one or a few cases to a hypothesis ■ Could inform practitioners about the type of intervention to try with future clients ○ Allow generalization from a case to an innovative intervention possibility ■ If effective for some with low risk and low cost, worth trying ○ Demonstrate operationalizing and measuring important variables ○ Predictions can be made about performance improvements over time and their durability ○ Serve as pilot projects to justify further research ○ Exposes ineffective interventions to inform practitioners ○ Collaboration between practitioners and academics
Poster Session
● Components of the poster itself include ○ An abstract written in 25-50 words ○ The purpose or aim of the study ○ The methodology including subject selection, measurements, procedures, or treatment techniques ○ Results presented in graphic form ○ Summary of the conclusions ● In addition to the poster, a handout is usually available that may include all the component, plus summary of literature review and contacting info
Major purposes of testing in clinical research
● Establish baseline data ○ Experimental research, pre-tests ● Evaluate outcome or effectiveness of treatment procedure ○ Experimental research, post-tests ● Assess the degree of relationship between two variables ○ Correlational research ● Evaluate quality of healthcare or education progress for accreditation ○ Evaluative research ● Assess developmental landmarks ○ Developmental research ● Assess individual values, interest, or attitudes ○ Survey research ● Evaluate differences between groups ○ Correlational, experimental research ● Evaluate functional assessment ○ Screening target population
global/social/ecological validity
● Even clear and persuasive inferences from the experimental data of SCED, with their selected, operationalized outcomes, do not automatically translate to the actual contextualized, occupational life of the individual participant ● One way to improve external validity is to collect data in multiple settings and on several participants ● A final inference needs to be made, ultimately about whether the intervention holds promise for similar individuals in their own unique naturalistic settings
Organization
● Finally the author should examine whether the background info is represented accurately, whether ideas unrelated to the topic have been introduced, and whether the hypothesis or guiding statement(s) have directed the writing ● Each statement of results should be supported by data ● Differences or contradictions in findings compared with previous studies need to be discussed and when possible explained ● The whole research paper should be reviews for sequence of ideas
Discussion, summary, conclusions, and implications of the findings
● In the final section, the writer summarizes the findings and attempts to interpret them in the context of previous studies in the same field ● Conclusions regarding acceptance or rejection of hypotheses are stated in this section ● Although statements of generalization can be made in this section, one must be careful not to over generalize or overstate the significance of the findings ● The writer should be aware of the limitations of the study and state them, expressing any cautions research regarding generalization to populations not included in the sample ● Implications for further research, present practice, or policy changes are advanced along with the arguments for these changes ● The internal and external validity of the study should be discussed ● The discussion section of the research paper is the most difficult part to write ● The discussion section should be a critical and analytical summary of the findings than a superficial overview of the results ● Because this part of the research paper is the most frequently read, it is crucial that the researchers summarize the results and critically review these findings in the context of prior studies ● The usefulness of the research paper, despite the findings, can be determined by the way in which the discussion section is written
Reference list and appendices
● In the last selection of the research paper, the writer lists all of the cited _________ ● The writer must be careful not to leave out any reference, to include all information in the citation, and to check for accuracy and correct spelling of information ● Sometimes the write wishes to place a bibliography and a reference list into the research paper ○ a bibliography contains any article, book, or chapter that might clarify the topic ○ A reference list includes journal articles, books, or other sources used in the research and preparation of the paper and cited with the article ● Any information (copy of the survey or test protocol, drawing of specific apparatus) can be placed in an appendix following the list of references
Operationalization of variables
● Just as a precise description of protocols in group experimental studies is crucial for future replications, so it is for SCED ● This precise definition of the applied intervention and of how the outcomes are measured is called ____________
When is an SCED applicable?
● Maintaining a baseline period is ethical and can be done long enough to establish a relatively stable picture of the typical range of performance on the dependent variable for this individual before intervention begins ● The desired outcome performance is measurable ● Repeated measurements of the individual are feasible and not contraindicated by limited endurance or a measurement tool that is affected by repeated administration ● Intervention creates an immediate or a near immediate effect on an outcome ● The case can be followed long enough to establish a persuasive pattern in the data ● Constructs of interest (whether occupation based or client centered factors) are feasible to quantify
Making revisions
● Once the research paper has been written, _____________ will be necessary- revising the research paper 5 or 6 times is not uncommon ● Each time the writer reviews the research paper for accuracy and cohesiveness, he or she may discover additional ways for revision or clarity ● The function of revision is to make sure specific editing requirements have been followed
Preparation for Writing
● One hindrance to the completion of the writing task is the failure to actually sit and write; however, allowing frequent interruptions during the actual writing period is equally problematic ● The first task in the preparation for writing is to get oneself into the proper frame of mind- this includes taking care of "settling activities" that led to interruptions ○ Ex: planning a large block of time, getting one's coffee or drink, assembling all the materials needed, arranging for child care, putting cat/dog outside, taking care of needed personal toiletries ● Clear and simple language is the hallmark of good scientific writing, and its most important quality is objective self-criticism ● For some individuals, a major block to the task of writing is the initial step of organizing their ideas on paper ● The place and time in which one writes are important initial considerations ○ Whatever the method and place, one must be aware of the environment that best facilitates creative writing ● Another consideration for writers is time schedule ● Writing a research paper or article is unlike casual writing in which an individual can use spare minutes throughout the day to complete the task ○ Scientific writing take mental energy and therefore requires concentration effort ● A third consideration consists of assembling the materials to be used before beginning any writing task ● Writing a book, journal article, or thesis requires self-discipline- the individual should prepare for writing as much as an athlete prepares for a sporting event ○ A period of conditioning and mental rigor prior to writing is important ● After the 1st draft, there will be ample time for revision, which in most cases will involve excision ● For the author or research investigator, the complete document is analogous to giving birth ○ Writing is a continuous process with much editing and revision, but one should be careful not to abandon the initial efforts prematurely ● The purpose of the research paper includes an explanation of why the question was formulated and how the research methodology serves to answer the question ○ In the final project (research paper), the writer summarizes the literature, discusses the finding-in this way, the final scientific paper becomes a part of the body of literature, placing the research findings within the context of previous works ● A well-conceived research paper requires active contemplation by the writer ○ The conceptual relationships between previous research and present findings must be considered ○ The hypothesis or guiding question(s) proposed by the researcher must be concisely and clearly stated and should cover all the points covered in the paper ○ The summary and arguments used in the paper to advance one's theory must be compared with viewpoints held by established researchers ■ A researcher may present findings that are contradictory to accepted beliefs in the scientific community ■ Under attack, the researcher must be able to defend the validity of the research findings ● Another essential component for writing a research paper is to understand the literature encompassing the topic as fully as possible- accomplished by an exhaustive literature review ○ Much more literature has been reviewed than will be discussed in the paper- this does not minimize the need for an extensive literature review; rather it emphasis the need for the researcher to understand and master the relationship of a specific literature within the context of a body of knowledge ○ It is not uncommon to revise the initial hypothesis or guiding question several times as one's knowledge base increases through reading and reviewing the literature ● Writing and reviewing literature is like an organic process that helps researchers modify their thinking while extending their knowledge base- it is a creative process that allows one to be objective and self-critical. This process is sometimes referred to as: ○ cognitive dissonance- the state in which there is conflict between one's attitudes and one's behavior, generally resulting in changing one's thinking so that there is equilibrium between the two ○ Reflective decision making- the process of making decisions by critically examining all sides of the issue ○ Socratic method- learning by raising questions ○ All these methods rely on the individual's ability to critically question and rethink a body of knowledge
Key points to presenting a paper
● Rehearse the presentation, considering time available for the presentation ● Consider the # of people & size of the room when preparing the paper. If the presenter has only 10 to 15 mins, discussing 1 or 2 major points is more feasible than a presentation of 90 mins ● Consider who will be in the audience- plan accordingly ● If the paper must be read verbatim, use conversational expression and eye contact ● When presentations are more than 15 mins long, prepare a visual presentation, such as Microsoft PPT presentation that can be used to discuss the paper rather than reading ● Use 3-x-5 cards to organize & remember the key points to present ● Allow 5-10 mins for questions ● Speak loudly enough for everyone to hear ● Prepare the visual presentation so that people in the last row can see them-if possible provide handouts of the overheads ● When preparing PPT presentations be sure that the print is large (fonts 36 or above) & that students have handouts of the presentation- a light print with a dark background is easier to read- a sans serif font (arial) is also preferred- animations may distract from presentation ● Be use equipment works prior to presentation- check 10 mins before presentation ● Include a beginning, middle, and end to your presentation- the beginning should establish rapport and set purpose & the ending should summarize the presentation-make conclusion memorable ● Adult learners are pragmatic, eager to apply what they learn, have a variety of experiences, and often challenge the presenter ● Include a 1-pg handout with references
Ethical considerations in testing
● Researcher publishing a new test should provide reliability, validity, normative data, scoring procedure, and qualifications for using the test ● Standardized test procedure can be carefully followed by tester ● Results should be reported that can be compared to specified populations ● Researcher using psychological tests should obtain informed consent for the subject and assure confidentiality
Conference Paper
● Researchers are encouraged to present the results of their findings at local, state, national, and international conferences ● When presenting a paper, it is important to recognize that those in attendance are adult learners with a heterogeneous knowledge base
Key components of preparing a poster
● Size of print: the print size should be large enough for participants to read it at a distance of 4 ft, generally a font size of 18-24 points ● Layout: the layout is determined by the space provided at the conference and directions from the conference committee. The material should be arranged to facilitate understanding of the material, generally top to bottom in a left-to-right sequence ● Amount of information: information should be written in bullets (3 to 4 word phrases) rather than in sentences ○ Include essential information about the design, procedure, and results ● Color: use color in highlighting the results and conclusions. A light color (yellow) printed on a dark paper is frequently easier to read ● Background: use of contrasting color or border behind the individual components will make the poster visually attractive
qualifications to administer a test: level Q
● Test can be purchased by individuals with one of the following backgrounds as determined by the particular purchase, along with formal training in the ethical use, administration, and interpretation of standardized assessment tools and psychometrics ○ Q1: degree or license to practice in the healthcare or allied healthcare field ○ Q2: formal supervised mental health, speech/language, and/or educational training specific to working with parents and assessing children, or formal supervised training in infant and child development
qualifications to administer a test: level C
● Test require a high level of expertise in test interpretation and can be purchased by individuals with ○ Licensure or certification to practice in your state in a field related to the purchase ○ Doctorate degree in psychology, education or closely related field with formal training in the ethical administration, scoring, and interpretation of clinical assessments
Methods
● The 3rd section of an experimental research paper includes a discussion of the ___________used in the study- this portion of the research paper should be written in a concise and succinct manner, with enough info available so that others can replicate ● Traditionally, there are 3 subdivisions in the __________ section: (a) participants, (b) the apparatus or tests, and © the procedures used for data collection and analysis ● Demographic information such as number of participants, ages, gender, and ethnicity is crucial- other include educational level, socioeconomic level, handedness, disabilities, prior testing, and previous need for therapies, especially OT ● If a control group is used, the demographics of the control group must be contrasted with the experimental group ● The manner in which subjects were chosen is also described in this section ● A description of the tests or apparatus includes the names of the tests or surveys used, any laboratory equipment needed, and any specific directions or adaptations to the test or procedures that might not be found in a test manual ○ When a standardized test is used, the reliability and validity of the test (or test portions) used must be indicated ○ If a survey or questionnaire has been developed for the study, a copy is put into the appendix following the major divisions of the paper ● Finally, the researcher describes the ________ of data collections and the statistical procedures used to analyze it
Results
● The 4th section of the research paper presents the __________and statistical findings from the study without attempting to interpret them ● Each hypothesis is discussed separately, with statistical interactions and main effects reported ● Tables and figures depicting significant relationships between variables should be self-explanatory and add to the info in the text rather than add new info
when is an SCED not applicable?
● The baseline measures show great instability ● It is not feasible, ethical or valid or safe to measure the dependent variable repeatedly ● There is a long or unknown lag time between the intervention and a demonstrable effect ● An interaction of the intervention studied with other simultaneous interventions is present ● Uncontrollable outside events or dramatically changing external conditions occur ● Only a brief period is available for intervention and measuring its effect (except for SCEDs using the ATD) ● The selected occupation-based constructs are difficult to measure through repeated data collection
Strengths of an SCED
● The intervention usually takes place in naturalistic settings ● The intervention is individualized, which accords with client centered practice ● In some SCEDs, the intervention is graded up or down if necessary ● They offer the possibility of enlisting multiple cases or collecting data in multiple settings or under multiple conditions to reveal how variables may interact ● An SCED can reveal the cumulative and complex interactions among all variables that are measured, giving rise to new insights, hypotheses or theories ● A strength of SCEDs is their ability to display changes in the level of the dependent variable over time (slope), and in the variability of the dependent variable during control (A) and experimental (B) phases ● This detail allows for a more sophisticated investigation of the relationship between the independent and dependent variables in the context of existing knowledge about them ● Historically, SCEDs have been criticized for lacking generalizability ○ Of course, as an N-of-1 study, an SCED cannot have the type of statistical generalizability that a group experiment can have when its sample has been randomly selected from a population ○ Such a claim is rarely made by the authors of an SCED ○ However, these authors themselves underestimate the strength of generalizability that an SCED can have ● One therapist achieving a positive outcome with one client using a specified intervention demonstrates that it is at least a possibility that other therapists can achieve this same result by applying the insights gained ● The knowledge that a possibility exists can be particularly valuable for intervention decisions for clients who have a rare condition, multiple diagnoses, unusual impairments or challenges, or for whom standard approaches have not been successful ● Some SCEDs (e.g., ATD) offer a fast reliable comparison of equipment options, tool options, postural or technique options, or options in environmental conditions or modifications, allowing for a persuasive argument to be made for the superiority of one over another ● An SCED can suggest why an intervention works with one individual and not with another, or which intervention works better for which individuals ● This aspect of discovering knowledge for customized therapy is often not pursued in group experimental studies
Clarity of thought
● The scientific writer should try to communicate ideas simply and rigorously ● It is not a sign of intellectual prowess to present ideas so that they are difficult to understand ● 6th reason for revision is to ensure that the paper is written clearly ● Scientific writing is prone to abstruseness and ambiguity, especially in clinical areas where language is used loosely without precise operational definitions ● One method of ensuring _______is to include a conceptual or operational definition for unusual terms or for terms specific to a certain discipline ● Even if the writer choose not to create a glossary, uncommon terms must be defined operationally ● It is not unusual for readers to look at charts, tables, or figures before reading the text, especially when the visual aids provide an overall summary of the principle finding
Review of the literature
● The second part of the research paper, which contains the ______________, is to some extent a measure of what the student knows about the subject ● The purpose of this section is to further introduce the reason for the study by reviewing previous research for the study ○ The researcher must presume that the reader has some knowledge in this area- the major studies must be discussed ● Some articles will be especially important to build a strong argument for the research design, and the a critical analysis of these articles will be an integral part of the literature review ● Research literature and theory need to be reviewed and discussed objectively, citing both positive and negative research findings that relate to the present research question- in this way, one can avoid research bias
Applying criterion referenced concepts to research: look at the following factors-
● Theory underlying concepts is identified ● Research evidence supporting any assumptions of the test are stated ● Content domain of test that considers a comprehensive view of skills, interests, and behaviors is discussed ● Specific target population is operationally defined in terms of age, intelligence, education, and degree of disability ● Test items are generated by selecting representative samples of behavior ● Performance standards are based on systematically collected data from representative samples of target population ● Test items are pilot studied for clarity and ease of admin ● Scoring methods are operationally defined ● Reliability and validity documented ● Test manual for admin, scoring and interpretation ● Degree of competency in admin is included
Weaknesses of an SCED
● There is a challenge with external validity ● SCEDs do not have the statistical models of the traditional group experimental designs with adequate degrees of freedom and statistical power to support internal validity ● The influence on the outcome of therapeutic relationship between the independent variable provider and the client may be difficult to assess ● Practitioner bias should be addressed in SCEDs but is difficult to control in this design ● Measurement limitations may also exist ● Although not exclusive to SCEDs, these consist of floor and ceiling effects, insensitivity in the dependent variable to small but important changes, an unknown relationship of the outcome performance or behavior to norms in society, or an unknown amount of the dependent variable that would be the criterion for independent performance in a population ● In the analysis of SCED data, there is often a bending of certain statistical assumptions, especially independence among data points ● That data display of SCED is vulnerable to graphing contingencies, such as missing data ● A dependable argument for causal relationships between the independent and the dependent variables can be complicated ○ It is often built upon several component inferences, unlike in a group experiment ○ In the latter, rejecting the null via a single statistical test can be taken as a sufficient reason to confidently claim that the intervention caused the better outcome in the clients receiving it, compared to the control group ○ Due to the possible honeymoon, or short term Hawthorne effect in the client upon receiving a novel treatment, SCEDs need to be replicated to strengthen any causal claim ● SCED researchers have recently issued the SCRIBE guidelines to protect against some of the possible flaws in SCED and to provide better detail in transparency so that replication by other researchers is more straightforward
Conceptual model for selecting a test instrument for clinical research- researcher must ask the following:
● What is the target population? ○ Ex: Typical children ● What are the specific areas of function to be measured? ○ Ex: Social-skills ● Where will the client be assessed? ○ Ex: Hospital ● What are the methods for evaluating function and how will the individual be assessed? ○ Ex: Observation of performance, Standardized test measuring function ● Is the test reliable and valid? ○ Ex: are there standardized directions for administration ● Are test results easily interpreted? ● What is the scale of measurement ○ Ex: Interval ● How will the individual be tested Ex: Small group
Writing the first draft
● Writers begin the ____________ after completing the outline- the outline aids inorganizing the literature review, but the outline is not set in stone ● After the __________has been written , the organization may need to be revised (can be several times) ● Writers should compose their initial thoughts with the expectation of revising them later ● A difficult skills for inexperienced writers is the ability to use notes effectively ● There is a tendency to quote extensively rather than paraphrase the author's words-this is caused by a lack of understanding about what the author meant ● Although quotes from the original source can be an effective means of relating information, too many quotations make the research paper difficult to read and leave "the impression that students have done cut and paste from books and articles they have read" ● Extensive quoting error and forming the habit of checking the APA manual ● It is better to quotations sparingly, intermixing the quotations with summaries and paraphrases of the original sources ● The literary and scientific styles used for citation often are determined by a particular discipline ○ the latter style is used most frequently in psychology, education, and social and health sciences
Abstract
● Written last and then placed at the beginning of the study, contains summary statements of each independent unit or section (the problem, literature review, research design, methods, results, discussion, conclusions, limitations of the study and recommendations for further research) ● Should contain enough information so that the reader will know the purposes of the study, the specific variables or groups investigate, the # of participants, the measuring instruments used, the results, and the conclusions presented ● Usually limited between 150-300 words, when preparing, it is useful to extract key sentences from each part of the study and then integrate the sentences through transitional phrases ● Extremely important to investigators reviewing literature, therefore vital to present as complete a summary of the total study as possible within the # of words permitted ● ____________are not a summary or discussion of the results as perceived by some investigators ● Should be a complete statement representative of the total study
qualifications to administer a test: level B
● test can be purchased by individuals with ○ Certification or full active membership in a professional organization ■ Ex: AOTA ○ A masters degree in psychology, education, OT, SLP, social work, or field closely related to the intended use of the assessment ○ Formal training in ethical administration, scoring, and interpretation of clinical assessments
criterion referenced tests (CRTs)
● tests based on a standard of performance, competence, or mastery, rather than comparison to a normative group. Based on socially acceptable minimal standards of performance
Norm-referenced tests (NRTs)
● used to assess degree if achievement, aptitude, capacity, interest, or attitude compared to established norms or standard scores based on a population
