EBP: Validity in Experimental Design
What is co-variation of cause and effect?
-*outcome* only is present with the *experimental effect* or the *degree* of outcome is related to the experimental *intensity* the *magnitude* of intervention
What is Maturation?
-1) a second threat to internal validity concerns processes that occur simply as a function of the *passage of time* and that are independent of external events 2) maturation may cause subjects to respond differently on a second measurement because they have grown older, stronger, healthier, more experienced, tired, or bored since the first measurement
What are Possible Solutions to Threat: Assignment?
-1) adequate defined inclusion and exclusion criteria random 2) assignment to groups 3) statistical adjustment
What is Violated Assumptions of Statistical Tests?
-1) all statistical tests are based on a variety of assumptions about the experimental data and the sample from which they are collected 2) if these assumptions are not met, statistical outcomes may lead to erroneous inferences 3) if you do a Student-T test on a non-standard population, your results will be invalid (Chi-square is used here) 4) if you use a Pearson correlation on ranked data you may get a lower correlation that using Spearman's rho
What is Assignment?
-1) assignment is a threat to research validity that occurs when the process of *placing* subjects into groups results in differences in baseline characteristics between or among the groups at the outset of a study 2) characteristics commonly of interest in clinical studies include subject age, gender, ethnic and racial background, education level, duration and severity of the clinical problem, presence of comorbidities, medication regimen, current functional status, and a current activity habits, among others
What falls under Length of Follow-Up?
-1) becomes a potential problem when a study addresses responses over time 2) if the time is too short, cannot make generalizations about long-term effects of treatment 3) conclusions and interpretations cannot be made beyond the scope of the study 4) the timeframe used for data collection must be considered when looking at the construct validity of the IV
What is Attrition?
-1) clinical researchers are often faced with the fact that subjects *drop out* of a study before it is completed 2) also called experimental *mortality*, is of concern when it results in a differential loss of subjects, drop outs that occur for specific reasons related to the experimental situation
What is Instrumentation?
-1) effects are concerned with the *reliability of measurement*. Observers can become more experienced and skilled at measurement between a pretest and posttest. 2) changes that occur in calibration of hardware or shifts in criteria used by human observers can affect the magnitude of the dependent variable, independent of any treatment effect
What are Single Group Threats To Internal Validity?
-1) history: confounding effect of specific event 2) maturation: effect is due to passage of time 3) attrition (aka experimental mortality): subjects drop out 4) testing: repeated testing; reactive measurements 5) instrumentation: reliability of tools used 6) statistical regression to the mean Research design of using a control group can be used to rule most of these out
What are Construct Validity of Causes and Effects?
-1) importance lies in operational definitions 2) a study can be internally sound but have no practical application 3) potential biases can be introduced into a study by the researchers or the subjects
What are Design strategies for inter-subject differences?
-1) inclusion criteria for homogeneity 2) blocking variable, eg age, education, etc 3) matching subjects 4) using subjects as own control 5) analysis of covariance (ANCOVA): theoretically removes the confounding influences of extraneous factors
What variables of interest will impact validity?
-1) independent 2) dependent 3) extraneous (influence of unrelated factors) when not controlled we call this a *confounding influence*
What does a *true* experimental study require?
-1) independent variable manipulated by the researcher 2) randomly chosen and assigned subjects 3) a control or comparison group
What are Threats To External Validity?
-1) interaction of treatment and selection: becomes a problem if samples are confined to certain types of participants 2) interaction of treatment and setting 3) interaction of treatment and history: cannot generalize results to different periods of time in the past or future 4) reactive or interactive effects of testing: a pretest makes a participant more aware or sensitive to upcoming treatment
What should you consider with Threats to Validity?
-1) is there a relationship between the independent variables and dependent variables? 2) given that a relationship does exist, is there evidence that one causes the other? 3) given that a cause and effect relationship is probable to what theoretical constructs can be the results be generalized? 4) can the results be generalized to persons, settings and times that are different form those employed in the experimental situation?
What are Possible Solutions to Threat: Compensatory equalization of treatment?
-1) mask the investigators to prevent knowledge of group assignment 2) ask all study participants to avoid discussing their activities with anyone
What are Possible Solutions to Threat: Diffusion of Treatments?
-1) mask the subjects to prevent knowledge of group assignment 2) keep subjects separated 3) ask all study participants to avoid discussing their study activities with anyone
What are Possible Solutions to Threat: Compensatory Rivalry or resentful Demoralization?
-1) mask the subjects to prevent knowledge of group assignment 2) mask the investigators to prevent knowledge of group assignment
What is no plausible alternative explanations?
-1) no other explanation for the response 2) confounding variables present threats to internal validity because they offer competing explanations for the observed relationship between the independent variables and dependent variables 3) these are considered as single group, multiple group, and social threats
What is Hawthorne Effect?
-1) participants experience change as a result of simply being in a research project 2) patients who were told preoperatively that they were part of a study scored significantly better on post op measures of knee pain and psychological well being compared to patients who were given the same information
What is Multiple-Treatment Interaction (Carry over effects)?
-1) participants receive more than one treatment or measurement 2) effects of previous treatment or measurement may influence subsequent ones 3) carryover or combined effects can occur making it difficult to interpret the results out of the context of several responses
What are Possible Solutions to Threat: History?
-1) random assignment to groups 2) timing of study to avoid the events 3) statistical adjustment
What are Possible Solutions to Threat: Maturation?
-1) random assignments to groups 2) time the study to minimize the effect 3) randomize technique or treatment order 4) statistical adjustment
What is History?
-1) refers to the confounding effect of *specific events*, other than the *experimental* treatment, that occur after the introduction of the independent variable or between a pretest or posttest 2) for example, a study examining the effect of education program for increasing the use of seat belt by teenagers. If the state passes a law mandating seat belt use during the course of the study, that represents a *history effect*
What is Construct Validity of Causes and Effects?
-1) refers to the theoretical conceptualization of the IV and DV a) how did the researcher think (conceptualize) the relationship b) if we study exercises to improve gait, we must conceptualize these variables in term of specific activities and measurement tools c) outcomes will be interpreted differently, for example if we study isometric knee exercise and measure gait speed than if we study walking as an exercise and measure distance walked 2) concerns the researcher's goals and how well experimental results can be generalized
What is Regression toward the Mean?
-1) regression is also associated with reliability of a test. When measures are not reliable, there is a tendency for *extreme scores* on the pretest to regress toward the mean on the posttest. This effect occurs even in the absence of intervention. 2) extremely low scores tend to increase; extremely *high scores* tend to regression is of greatest concern when individual are selected on the basis of extreme scores
What are Possible Solutions to Threat: Attrition?
-1) replacement if appropriate and feasible 2) statistical adjustment
What are Possible Solutions to Threat: Instrumentation?
-1) select appropriate technique 2) user training and practice 3) specific protocols for use by testers 4) instrument calibration
What are Multiple Group Threats To Internal Validity?
-1) selection-history effects: result when experimental groups have different experiences between the pretest and posttest 2) selection-maturation effects: occur when the experimental groups experience a maturational change at different speeds 3) Selection-attrition effects: 4) Selection-testing effects: occur when pretest affects groups differently 5) Selection-instrumentation interaction: occur when the test is not consistent across groups 6) Selection-regression: is of concern are specifically divided based on higher and lower pretest scores Selection interaction threats occur when multiple groups are used and the groups are not equal.
What are Possible Solutions to Threat: Testing?
-1) specific protocols, including scripts for use by testers 2) provide subjects with practice sessions
What are (4) categories of threats?
-1) statistical conclusion validity 2) internal validity 3) construct validity of causes and effects 4) external validity (normally, these threats are considered sequentially from top to bottom when analyzing a study results)
What is Reliability and Variance?
-1) statistical conclusions are threatened by any extraneous factors that increase variability within the data, such as unreliable measurement, failure to standardize the protocol, environmental interferences, or heterogeneity of subjects 2) extraneous factors that increase variability within the data will give unreliable measurements, or an inability to standardized data
What are (3) components of causality?
-1) temporal precedence 2) co-variation of cause and effect 3) no plausible alternative explanations
What is Testing?
-1) testing effects concern the potential effect of *pretesting* on the dependent variable 2) the mere act of collecting data changes the response that is being measure 3) testing effects can refer to improved performance or increased skill that occurs because of familiarity with measurements
What is Power?
-1) the ability of a statistical test to *reject* the null hypothesis, that is to document a real relationship between independent and dependent variables 2) power calculations are done *before* the study begins to see what *sample size* is needed and then after results are in to show the strength of the null hypothesis rejection (or inability to reject)
What are Operational Definitions?
-1) the belief that most constructs require use of multiple experimental interventions and multiple measurement methods 2) if these are only one dimensional the results will only apply to a limited aspect of the construct 3) multiple-treatment interaction can create carryover or combined effects 4) order effects can occur if experimental interventions are consistently given in same order
What is temporal precedence?
-1) the cause precedes the effect 2) we must be able to document that the cause precedes the effect; that is any change in outcome must be observed only after treatment is applied
What is External Validity?
-1) the extent to which the results of a study can be generalized beyond the internal specifications of the study sample 2) concerned with the usefulness of the information outside the experimental situation 3) the generalization of a study is primarily related to the specific patient context and conditions under investigation
What is Error Rate?
-1) with certain tests, the probability of drawing incorrect conclusions increases as the number of repeated tests increase. Statistical procedures are generally available to control for this treat 2) measurement error 3) too high standard deviation 4) statistical tests are normally done to overcome these
What is Internal Validity?
-Causality: 1) determines if experimental treatment really caused the observed change in the DV 2) extraneous factors interfere with cause and effect inferences 3) randomization and control groups help avoid extraneous factors
What is Experimental Bias?
-biases introduced because of participant or researcher expectations
What is control?
-in the design will limit the impact of extraneous factors
How can you rule out threats to internal validity?
-many threats can be ruled out by the use of *random assignment* and *control groups*
What happens when the internal validity is increased?
-normally, when internal validity is increased, external is lowered and vice versa
What is Avis Effect?
-participants in control group try harder simply because they are in the control group
What is Statistical Conclusion Validity?
-refers to the appropriate use of statistical procedures for analyzing data
What are Social Threats To Internal Validity?
-refers to the pressures that can occur in research situations that may lead to differences between groups 1) Diffusion or Imitation of Treatments: experimental groups socializes with controls 2) Compensatory Equalization of Treatments: researcher is not blinded to intervention and tries to compensate 3) Compensatory Rivalry: subjects try differently. When one group's assigned treatment is perceived as more desirable than the other's; subjects receiving the less desirable treatment may try to compensate by working extra hard to achieve similar results 4) Resentful Demoralization: subjects receiving less desirable treatments may be demoralized or resentful. Their reactions may be to respond at lower levels of performance
What is Rosenthal Effect/experimental effect?
-researcher's behaviors or appearance impact how the participant responds
How do Social Threats result?
-result through the interaction of the subject and investigators. Blinding is desirable.
What are Threats to Statistical Conclusion Validity?
-statistical conclusion validity concerns the *potential inappropriate use of statistical data*, leading to *invalid conclusions* about the relationship between independent and dependent variables 1) low statistical POWER 2) violated assumptions of statistical tests 3) error rate 4) reliability and validity
What is external validity?
-the degree to which results of a study can be *generalized* to persons or settings outside the experimental situation
What is internal validity?
-the degree to which the *relationship* between the *independent* and *dependent* variables is free from the effects of extraneous factors
What is validity?
-the overall amount of control from all sources in an experimental study
What is research validity?
1) research validity related to the *truthfulness* or *accuracy* of a study's results and is determined based on how well the research design elements: a) focus on what the investigator wants to know b) prevent unwanted influences from contaminating the study's outcomes
