Quiz #7

True Experimental Designs (3) - Notes

*KNOW FOR QUIZ and EXAMPLES! 1. Between Groups Factor or independent variable - Experimental subjects different than control subjects - EX: Males vs Females 2. Within Subjects Factor or independent variable - EX: One group of subjects getting both experimental and control conditions 3. Mixed Design includes between groups and within subjects factors (factorial design - more than 1 factor) - EX: Experimental vs control groups (between) - EX: Trial 1 vs trial 2 (within)

Control, Realism, Internal and External Validity - Notes

*KNOW FOR QUIZ! - look at diagram on slide

MAXICON Principle - Notes

- "MAX"imize experimental variance - m"I"n error variance → reliability and validity - "CON"trol extraneous variance

Experimental Design - Symbols - Notes

- "R" = random assignment (true experimental design) - "O" = observation or measurement of dependent variable - "T" = treatment or manipulation of independent variable

Factorial Designs - Notes

- *GRAPH COULD BE ON QUIZ! → What is the factorial design? → Could also be a table or bar graph - *review slide and notes - 3 by 3 → pretest, posttest 1, posttest 2 → "3" lines - mixed design study

Switched Replication Design - Notes

- *review slide - 4 groups (between groups study) - 5 trials - Group 1 gets intervention "T" = pretested, then treatment, then posttested 4 times - Group 2 gets intervention "T" = posttested 3 times - Group 3 gets intervention "T" = posttested 2 times - Group 4 gets intervention "T" = posttested 1 time - randomly assigned = true experimental design - intact groups = quasi experimental - improved

Time Series Design - Significant Effects - Notes

- *review slide - 8 measures - after first 4 measures, intervention, then look at next 4 measures, analyze results - the effect occurred, but it did not last (6, 7, 8) - improvement occurring (1, 2, 3, 4) - rate of improvement changes = accelerated (4, 5)

Main Effect: Interaction - Notes

- *review slide - lines no longer parallel (lines intersect = picture of interaction) - *review next slide and notes - marginal means are the same (A1 = A2) and (B1 = B2) - cells means are different: → A by B interaction with no main effect → A1 = A2 and B1 = B2, but real difference is cell A1B1 = A2B2 < A2B1 = A1B1

Reversal Design: No Significant Effects - Notes

- *review slide - mean performance drops between 2 and 3 - significant improvement trial 2 and 3 and "E" and "F" - intervened again "T2" = performance dropped in "D", improved in "E" and no effect in "F" (not significant, failed to replicate)

Nonequivalent Control Group - True Factor Mixed Design - 2 (T/C) by 2 (Pre/Post) - Notes

- *review slide - no level of randomization - O1 = O3 (pretest) - "r" = randomly assign intact (pre determined) groups to conditions - what is this design called in epidemiology? → community trials (with randomization to groups)

Reversal Design: Significant Effects - Notes

- *review slide - require multiple measures - for all 3 trends, there is improvement between 2 and 3 when intervention occurs - intervene again "T2" = replication built in → improvement occurred again = 4 and 5

Time Series Design: No Significant Effects - Notes

- *review slide - the intervention (after trial 4) no effects - "F" = improvement but at same rate before intervention - "G" = no difference in improvement (1, 2) and (4, 5) and (7, 8)

A - Main Effect - No Interaction - Notes

- *review slide and notes

Three Factor Effects for Significant Testing - Notes

- *review slide and notes - Main Effects: → A → B → C - Two Factor Interaction Effects: → A by B → A by C → B by C - Three Factor Interaction Effect: → A by B by C *use statistical technique: - analysis of variance (ANOVA) → analyzing one dependent variable - multivariate analysis of variance (MANOVA) → analyzing 3 dependent variables

Experimental Design: An Example

- *review slide and notes - design: 2 x 2 x 2 - A = type of training - INT vs. LSD - 2 levels - B = days per week - 3 vs. 5 - 2 levels - C = duration - 30 minute vs. 30 minute - 2 levels

No Main Effect: Interaction - Notes

- *review slide and notes - lines not parallel (interact)

Main Effect - No Interaction - Notes

- *review slide and notes - lines parallel to each other (picture of no interaction) - "A2" lower than "A1" (A1 > A2) - *review next slide and notes - A by B interaction with FALSE main effects → A1 > A2 and B2 > B1, but real difference is cell → A2B1 < other cells; interaction is an "inconsistent difference"

Table 18.3 - Quasi Experimental - Notes

- *review table - look at differences in internal vs. external validity

Table 18.2 - True Experimental Design - Notes

- *review table - look at differences in interval vs. external validity → + = strength → - = weaknesses → blank = not relevant → ? = questionable

Table 18.1 - Preexperimental Design - Notes

- *review table (no need to memorize, just recognize) - look at validity threats (for internal and external) → lots of negatives = weaknesses → blanks = not relevant → ? = questionable

Experimental Research - Notes - Part 1

- Experimental Research → the manipulation of an independent (treatment or experimental) variable to cause a measurable (1) and replicable effect in one or more dependent variables (2) while controlling extraneous variables (3)

Nonequivalent-Control Group Design - Text

- a design using a nonequivalent control group is frequently used in real world settings where groups cannot be randomly formed - the design is as follows: O1 T O2 O3 O4 - this is a pretest-posttest design without randomization - researchers compare O1 and O3 and declare the groups equivalent if this comparison is not significant - unfortunately, just because the groups do not differ on the pretest does not mean that they are not different on any number of unmeasured characteristics that could affect the outcome of the research - if the groups differ when O1 and O3 are compared, ANCOVA is usually employed to adjust O2 and O4 for initial differences - alternatively, a within (the pretest-posttest comparison - repeated measure) and between (treatment and control group comparison) two-way ANOVA could be used to analyze whether groups changed from pretest to posttest and whether the change was different for those in the treatment and control groups

One-Shot Study - Text

- a group of participants receives a treatment followed by a test to evaluate a treatment: T O - this design fails all the test of good research - all that can be said is that at a certain point in time this group of participants performed at a certain level - in no way can the level of performance (O) be attributed to the treatment (T)

Placebos #2 - Definition - Text

- a method of controlling a threat to internal validity in which a control group receives a false treatment while the experimental group receives the real treatment - used to evaluate whether the observed effect is produced by the treatment or is a psychological effect - frequently, a control condition is used in which participants receive the same attention from and interaction with the experimenter, but the treatment administered does not relate to performance on the dependent variable - EX: controlling psychological effects studying the use of steroids to build strength in athletes → to combat the fact that athletes may become stronger because they think they should when using steroids, a placebo (pill that looks like a steroid) is used - helps control: Hawthorne effect, expectancy effect, and halo effect

Double Blind #4 - Definition - Text

- a method of controlling a threat to internal validity in which neither the participant nor the experimenter knows which treatment the participant is receiving - EX: controlling psychological effects studying the use of steroids to build strength in athletes → the athlete, the person dispensing the steroid or placebos, and the testers do not know which group received the steroids - helps control: Hawthorne effect, expectancy effect, and halo effect

Blind Setup #3 - Definition - Text

- a method of controlling a threat to internal validity in which participants do not know whether they are receiving the experimental or control treatment → the participant is "blind" to the treatment - EX: controlling psychological effects studying the use of steroids to build strength in athletes → the athletes do not know whether they receive the placebo or the steroid - helps control: Hawthorne effect, expectancy effect, and halo effect

Quasi-Experimental Design - Definition - Text

- a research design in which the experimenter tries to fit the design to real world settings while still controlling as many of the threats to internal validity as possible - used in: kinesiology, physical education, exercise science, sport science and other areas - use randomization to control threats to internal validity (but can be difficult) - random assignment sometimes cannot be used in many settings → EX: if a researcher were studying the effect of an exercise program on the aged in a community setting, random assignment would not work because people select where to enroll in classes based on various factors of their lives (convenience, membership, transportation, etc.), not whether their choice helps a researcher. - Asking people to go to another site or to attend class at another time would probably reduce the number of people who agree to participate and increase participant attrition (or increase experimental mortality). - Quasi experimental designs never quite control internal validity as well as true experimental designs do, but they allow us to conduct investigations when true experiments cannot be used or when a true experimental design significantly reduces external validity

Single-Subject Design - Text

- a researcher is looking for the effect of an intervention on a single subject - sometimes called: N=1 designs (because they often have only one subject) - a researcher using this design is looking for the effect of treatment without using randomization - look at changes on graphs and do not analyze the results with statistics - most often used in clinical settings → EX: observing physical education instruction, pursuing sport psychology work with athletes, studying an outstanding performer (Olympic athlete) , or looking at the motor function of a person with a physical impairment (Parkinson's disease) - a participant in this type of study is typically measured repeatedly on the tasks of interest - many trials are needed for evaluating the influence of the treatment - during some periods a baseline measurement is obtained, and during other periods a treatment is administered - the focus is often on participant variability as well as average values - quasi experimental time series, reversal, and switched replication designs can work as single subject designs - when they are used as single subject designs, they are often called: A-B or A-B-A-B designs, where A refers to the baseline condition (no treatment) and B refers to when the treatment is administered - sometimes more than one treatment is administered to the same participant → counterbalancing the treatment order to separate treatment effects is important - *review graph on 18.3 - *review graph on 18.4

Ex Post Facto Design - Text

- a static group comparison but with the treatment not under the control of the experimenter - EX: compare characteristics of athletes vs. non athletes, highly fit vs. unfit individuals, female vs. male performers, and expert vs. novice performers → in effect, we are searching for variables that discriminate between these groups → our interest resides in the question "Did these variables influence the way these groups became different?" → of course, this design cannot answer this question, but it may provide interesting insights and characteristics for manipulation in other experimental designs → this design is also often called: "casual comparative design"

Avis Effect - Definition - Text

- a threat to internal validity wherein participants in the control group try harder just because they are in the control group

Sources of Invalidity - Text

- all experimental design types have strengths and weaknesses that pose threats to the validity of the research - fundamental is a distinction between internal validity and external validity - internal validity = the basic minimum without which any experiment is interpretable → Did in fact the experimental treatments make a difference in this specific experimental instance? - external validity = asks the question of generalizability → to what populations, settings, or treatment variables can this effect be generalized? - both internal and external validity are important in experiments, but they are frequently at odds in research planning and design - a more realistic approach is to identify the specific goals and limitations of the research effort, therefore, the researcher can then plan the research with one type of validity as the major focus while maintaining as much of the other type of validity as possible

Randomization #1 - Text

- allows the assumption that the groups do not differ at the beginning of the experiment - controls for history up to the point of the experiment; that is, the researcher can assume that past events are equally distributed among groups → this does not control for history effects during the experiment if experimental and control participants are treated at different times or places - the researcher must try and prevent an event (besides the treatment) from occurring in one group but not in the other groups - randomization controls for maturation because the passage of time is equivalent in all groups - statistical regression, selection biases, and selection maturation interaction are all controlled because they occur only when groups are not randomly formed - the matched pair technique matches pairs of participants who are equal on some characteristic and then randomly assigns each to a different group → the researcher may want very tight control on previous experience in strength training; then participants would be matched on this characteristic and then randomly assigned to the experimental and control groups - a matched group technique may also be used → involves non random assignment of participants to experimental and control groups so that the group means are equivalent on some variable → this procedure is generally regarded as unacceptable because the groups may not be equivalent on other unmeasured variables that could affect the outcome of the research - in within subject design, the participants are used as their own controls - this means that each participant received both the experimental and the control treatment → in this type of design, the order of treatments should be counterbalanced; that is, half the participants should receive the experimental treatment first and then the control, and the other half should receive the control treatment first and then the experimental treatment

One-Group Pretest-Posttest Design - Text

- although very weak, is better than the one-shot design - we can at least observe whether any change in performance has occurred: O1 T O2 - if O2 is better than O1, we can say that the participants improved - EX: Bill Biceps conducted an exercise test at a health club. Participants then trained 3 days per week, 40 minutes per day, at 70% of their estimated VO2 max, for 12 weeks. After the training period, participants retook the exercise test and significantly improved their scores. Can Mr. Biceps conclude that they exercise program causes the changes that he observed in the exercise test performance? → Unfortunately, this design does not allow us to say why the participants improved. → Certainly, the improvement could be due to the treatment, but it could also be due to history. → Some event other than the treatment (T) may have occurred between the pretest (O1) and the posttest (O2); the participants may have exercised at home on the other days. → Maturation is a rival hypothesis. The participants may have gotten better (or worse) just as a result of the passage of time. → Testing is a rival hypothesis; the increase at O2 may be the result only of experience with the test at O1. - If the group being tested is selected for some specific reason, then any of the threats involving selection bias can occur. - This design is most frequently analyzed by the dependent "t" test to evaluate whether a significant change occurred between O1 and O2.

True Experimental Design - Definition - Text

- any design used in experimental research in which groups are randomly formed and that controls most sources of invalidity - allows the assumption the groups were equivalent at the beginning of the research - this controls for past (but not present) history, maturation (which should occur equally in the groups), testing, and all sources of invalidity that based on nonequivalence of groups (statistical regression, selection biases, and selection maturation interaction) - only the experimenter can make sure that nothing happens to one group (besides the treatment) and not the other (present history), that scores on the dependent measure do not vary as a result of instrumentation problems, and that the loss of participants is not different between the groups (experimental mortality)

Reactive or Interactive Effects #1 - Text

- can be controlled only by eliminating the pretest - these effects can be evaluated, however, by only two designs: → pretest-posttest randomized groups → Solomon four-group design

Switched-Replication Design - Text

- can be either true or quasi experimental depending on whether levels are random or intact groups - *review table - if participants are randomly assigned to level 1 through 4, the design is a true experiment - if levels 1 through 4 are different intact groups (ex: tennis players in college, high school , and two age levels of youth leagues), then the design is quasi-experimental - any number of levels beyond two can be used, but the number of trials must be one greater than the number of levels - this design has two strong features: (1) the treatment is replicated several times and (2) longterm treatment effects can be evaluated - there is no standard statistical analysis for this design, but various ANOVAs with repeated measures could be used

Instrumentation #2 - Text

- cannot be controlled or evaluated by any design - only the experimenter can control this threat to internal validity - test reliability = the answers must be consistent - controlling this frequently involves the assessment of test reliability across situations, between and within testers or observers, and within participants - the validity of the instrument (does it measure what it is supposed to measure?) must also be established to control for instrumentation problems - the total process for establishing appropriate instrumentation is = psychometrics

Instrumentation #4 - Text

- changes in instrument calibration, including lack of agreement within and between observers - frequently faced in exercise science research → EX: suppose the researcher uses a spring loaded device to measure strength; unless the spring is calibrated regularly, it will decrease in tension with use; therefore, the same amount of applied force produces increases readings of strength compared with earlier readings - instrumentation problems also occur in research using observers - unless observers are properly trained and regularly checked, the same observer's ratings may systematically vary across time or participants (called observer drift), or different observers may not rate the same performance in the same way

Selection Bias #6 - Text

- choosing comparison groups in a nonrandom manner - occurs when groups are formed on some basis other than random assignment - thus, when treatments are administered, because the groups were different to begin with, always present is the rival hypothesis that any differences found are due to initial selection biases rather than the treatments - showing that groups were not different on the dependent variable at the beginning of the study does not overcome this shortcoming

Static Group Comparison - Text

- compares two groups, one of which receives the treatment and one of which does not: T O1 ---------- O2 - but we do not know whether the groups were not equivalent when the study began, as indicated by the dotted line between the groups - this means that the groups were selected intact rather than being randomly formed - we are thus unable to determine whether any differences between O1 and O2 are because of T or only because the groups differed initially - this design is subject to invalidity because of selection biases and the selection maturation interaction - A "t" test for independent groups is used to evaluate whether O1 and O2 differ significantly - but even if they do differ, the difference cannot be attributed to T

MAXimize - Notes

- develop the operational definitions of the independent variables to make the levels as different as possible - *review slide example → 2 (A) by 2 (B) = independent variables = 2 x 2 → A1 to be very different from A2 → B1 to be very different from B2 - Weinberg, et al., 1979 → confidence or self-efficacy in a persistence task → competing against each other (looked at levels or confidence with time) → high confidence VERY different from low confidence - Blocking designs → age to age group → Ex: young vs. old → 40 or less vs. 40+ → 40 or less vs. 50+ (better = no overlap)

Interaction in Factorial Design - Notes

- effects (hypotheses) to be tested - what hypotheses need to be tested? → A - main effect → B - main effect → A by B - interaction effect *review slide and notes

Matching - Notes

- equate research groups on extraneous variable → works well with one extraneous variable - *review example diagram on slide - groups can be pre tested and then matched so they are equal before study (example)

History #1 - Text

- events occurring during the experiment that are not part of the treatment - some unintended event occurred during the treatment period - EX: in studying the effects of a semester of physical education on the physical fitness of kids, if 60% of those kids participated in a recreational soccer program, this would constitute a history threat to internal validity (program likely to produce benefits to physical fitness that would be difficult to separate from the physical education program)

Chapter 18 - Experimental Research (Part 1) - Text

- experimental research attempts to establish cause and effect relationships - an independent variable is manipulated to judge its effect on a dependent variable - three criteria must be present to establish cause and effect: (3) 1. The cause must precede the effect. - EX: The starting gun in a race precedes the runners' beginning the race; the runners' beginning does not cause the starting gun to go off 2. The cause and effect must be correlated with each other. - just because two variables are correlated does not mean one causes the other; cause and effect cannot exist unless two variables are correlated 3. The correlation between cause and effect cannot be explained by another variable. - EX: the relationship between the academic performance of elementary school children and shoe size was explained by a third variable, age

Expectancy #9 - Text

- experimenters' or testers' anticipating that certain participants will perform better - the researcher anticipates that certain behaviors or results will occur - occurs usually where participants or experimental conditions are clearly labeled → EX: testers may rate skilled participants better than unskilled participants, regardless of treatment - this effect is also evident in observational studies in which the observers rate posttest performance better than pretest performance because they expect change - if experimental and control groups are identified, observers may rate the experimental group better than the control group even before any treatment occurs - the expectancy effect can influence participants too → EX: in a youth sport study, coaches may actually cause poorer performance by substitutes (compared with starters) because the substitutes realize that the coach treats them differently (the coach may show less concern about incorrect practice trials)

Controlling Threats to External Validity - Text

- generally controlled by selecting participants, treatments, experimental situation and tests to represent some larger population - random selection (or good enough sampling) is the key to controlling most threats to external validity → elements other than participants may be randomly selected → EX: the levels of treatment can be randomly selected from the possible levels, experimental situations can be selected form possible situation, and the dependent variable (test) can be randomly selected from a pool of potential dependent variables - questions to consider: → does the study have enough characteristics of real world settings that participants respond as if they were in the real world? → is ecological validity present? - the reactive or interactive effects of testing can be evaluated by the Solomon four groups design - interaction of selection biases and the experimental treatment is controlled by the random selection of participants - the reactive effects of experimental arrangements can be controlled only by the researcher - multiple treatment interference can be partially controlled by counterbalancing or randomly ordering the treatments among participants

Time-Series Design - Text

- has only one group but attempts to show that the change that occurs when the treatment is administered differs from the times when it is not - depicted as: O1 O2 O3 O4 T O5 O6 O7 O8 - the basis for claiming that the treatment causes the effect is that a constant rate of change can be established from O1 to O4 and from O5 to O8, but that this rate of change varies between O4 and O5, where T has been administered → EX: in figure 18.2, lines A, B, and C suggest that the treatment (T) results in a visible change between observations, whereas, lines D, E, F, and G indicate that the treatment has no reliable effect - regression techniques used for statistical analyses - this design appears to control for a number of the threats to internal validity → EX: maturation is constant between observations; testing effect can also be evaluated, although they could be difficult to separate from maturation; selection biases also appear to be controlled because the same participants are used at each observation; history, instrumentation and mortality are controlled only to the extent that the researcher controls them *A reminder about using intact groups in quasi experiments: The use of intact groups and, particularly, the timing of when a treatment is delivered to a group, influences the analysis and the number of groups needed for a study.

True Experimental Designs - Notes

- include randomization - not perfectly valid but enough validity to be convinced of a cause (T) and effect (O) relationship

Quasi-Experimental Designs (5) - Notes

- increase external validity - decrease internal validity - time series design - reversal design = doing an intervention, taking it away, and doing it again - nonequivalent control group design - switched replication design = intervention done differently across repeated measures - interaction in factorial designs

Experimental Mortality #7 - Text

- loss of participants from comparison groups for nonrandom reasons - even when groups are randomly formed, this threat to internal validity can occur → EX: participants may remain in an experimental group taking part in a fitness program because it is fun, whereas participants in the control group become bored, lose interest, and drop out of the study - also, the opposite can occur too → EX: participants may drop out of an experimental group because the treatment is too difficult or time consuming

Pre Experimental Designs Continued - Notes

- not acceptable for research purposes - rival hypotheses or explanations can explain results - you cannot say the manipulation of T caused an effect in O

Experimental Mortality #3 - Text

- not controlled by any type of experimental design - the experimenter can control this threat only by ensuring that participants are not lost (at all, if possible) from groups - can be handled in advance of research by carefully explaining the research to the participants and emphasizing the need for them to follow through with the project

Preexperimental Designs - Definiton - Text

- one of three types of research design that control few of the sources of invalidity and that do not have random assignments of participants to groups: → one-shot study → one-group pretest-posttest design → static group comparison

Maturation #2 - Text

- processes within the participants that operate as a result of time passing (ex: aging, fatigue, hunger) - most often associated with aging - occurs frequently in designs in which one group is tested on several occasions over a long period -EX: elementary physical education teachers giving a fitness test in the early fall and again in the late spring → maturation is a plausible rival hypothesis for the observed increase where the children have grown larger and stronger and thus probably run faster, jump higher and throw farther

Randomized-Group Design - Text

- resembles static group comparison except that groups are randomly formed: R T O1 R O2 - if the researcher controls the threats to internal validity that are not controlled by randomization, has a sound theoretical basis for study, and meets the necessary-and-sufficient rule, then this design allows the conclusion that significant differences between O1 and O2 are due to T - an independent "t" test is used to analyze the difference between O1 and O2 - this design represents two levels of one independent variable - it can also be extended to any number of levels of an independent variable: R T1 O1 R T2 O2 R O3 - here, three levels of the independent variable exist, where one is the control and T1 and T2 represent two levels of treatment - this design can be analyzed by simple ANOVA, which contrasts the dependent variable as measured in the three groups (O1, O2, O3) - EX: T1 is training at 70% VO2 max, T2 is training at 40% VO2 max, and the control group is not training → the variables O1, O2, and O3 are the measures of cardiorespiratory fitness in each group taken at the end of the training - this design can also be extended into a factorial design: that is, more than one independent variable (IV) could be considered - independent variable 1 (IV1) has three levels (A1, A2, A3), and independent variable 2 (IV2) has two levels (B1, B2) → this results in six cells (A1B1, A1B2, A2B1, A2B2, A3B1, A3B2) to which participants are randomly assigned → at the end of the treatments, each cell is tested on the dependent variable (O1 through O6) → this design is analyzed by a 3 x 2 factorial ANOVA that tests the effects of IV1 (Fa), IV2 (Fb, and their interaction (Fab)

Randomization (2) - Notes

- some level of randomization should occur in a true experiment 1. Between Groups Design - random assignment: → subject to groups (PEOPLE)* 2. Within Subjects or Repeated Measures Design - random assignment: → order of CONDITIONS* → counterbalance order (ex: Latin Square Design)

Statistical - Notes

- statistical analysis to control the influence of an extraneous variable on the relationship between an independent and dependent variable → EX: Analysis of Covariance → EX: Part or Partial Correlation → EX: Residual Scores → EX: Logistic Regression

Cause and Effect with Logic (7) - Text

- statistics does not establish cause and effect, they can only reject the null hypothesis and identify the percentage of variance in the dependent variable accounted for by the independent variable or the effect size - neither of these procedures establishes cause and effect (they are necessary but not sufficient) - cause and effect can be established only by applying logical thinking to well designed experiments - this process establishes that no other reasonable explanation exists for the changes in the dependent variable except the manipulation of the independent variable - the application of this logic is made possible by the following: (7) 1. Selection of a good theoretical framework. 2. Use of appropriate participants. 3. Application of an appropriate experimental design. 4. Proper selection and control of the independent variable (treatment). 5. Appropriate selection and measurement of the dependent variable. 6. Use of the correct statistical model and analysis. 7. Correct interpretation of the results. *Establishing cause and effect requires logical thinking applied to well designed experiments.

Testing #3 - Text

- the effects of one test on subsequent administrations of the same test - EX: if athletes are administered a multiple choice test to evaluate their knowledge about steroids today and again two days later, they will do better the second time even though no treatment intervened → taking the test once helps in taking it again - the same effect is present in physical performance tests, particularly if the participants are not allowed to practice the test a few times → EX: if a class of beginners in tennis attempts to hit 20 forehand shots delivered to them from a ball machine today and again three days later, they will usually do better the second time (they learned something from performing the test the first time)

Assimilation - Notes

- the extraneous variable could have some research interest - build it into the study → *review example diagram on slide

Statistical Regression #5 - Text

- the fact that groups selected based on extreme scores are not as extreme on subsequent testing - can occur when groups are not randomly formed but are selected based on an extreme score on some measure → EX: if someone rates the behavior of a group of children on a playground on an activity scale (very active to very inactive) and two groups are formed, one of very active children and one of very inactive children, statistical regression is likely to occur when the children are next observed on the playground; the children who were very active will be less active and the very inactive children will be more active (both groups will regress, move from the extremes and toward the overall average → this phenomenon reflects only the fact that a participant's score tends to vary about her average performance - extreme scores may simply reflect the observation of a performance on the high (or low) side of the participant's typical performance - the next performance is usually not as extreme, therefore, when average scores of extreme groups are compared from one time to the next, the high group appears to get worse, whereas the low group appears to get better - statistical regression is a particular problem in studies that attempt to compare extreme groups selected on some characteristic, such as highly anxious, fit, or skilled participants vs. not very anxious, fit, or skilled participants

Pretest-Posttest Randomized- Groups Design - Text

- the groups are randomly formed, but both groups are given a pretest as well as a posttest: R O1 T O2 R O3 O4 - the major purpose of this type of design is to determine the amount of change produced by the treatment; that is, does the experimental group change more than the control group? - this design threatens the internal validity through testing, but the threat is controlled because the comparison of O3 to O4 in the control group as well as the comparison of O1 to O2 in the experimental group includes the testing effect - thus, although the testing effect cannot be evaluated in this design, it is controlled - this design is frequently used in the study of physical activity, but its analysis is complex - three common ways to do statistical analysis of this design: → factorial repeated measures (ANOVA) → ANCOVA → obtain difference/gain score = a score that represents the difference (change) from pretest to posttest - in this design, an important question is: does one group change more than the other group? - this design can also be extended into more complex forms

Solomon Four-Group Design - Text

- the only true experimental design to specifically evaluate one of the threats to external validity: reactive or interactive effects of testing - the design is depicted as follows: R O1 T O2 R O3 O4 R T O5 R O6 - combines randomized groups and the pretest-posttest randomized group designs - the purpose is to determine whether the pretest results increases sensitivity of the participants to the treatment - this design allows a replication of the treatment effect: (is O2 > O4? and is O5 > O6?) - an assessment of the amount of change due to the treatment: (is O2 - O1 > O4 - O3?) - an evaluation of the testing effect: (is O2 > O5?) - thus, this experimental design is extremely powerful - unfortunately, it is also inefficient because twice as many participants is required - consequently, this design is in limited use - there is no good way to analyze this design statistically (best alternative is 2 x 2 ANOVA

Selection-Maturation Interaction #8 - Text

- the passage of time affecting one group but not the other in nonequivalent group designs - occurs only in designs in which one group is selected because of some specific characteristics, whereas the other group lacks this characteristic → EX: differences between 6 year olds in two school districts; students in one school district form the experimental group that receives a fitness program, and students in the other are a control group; if the school districts have different admission policies so that the 6 year olds in the experimental group are 5 months older, it would be difficult to determine whether the fitness program or the fitness program combined with the participants' advanced age produced the observed changes

Reactive or Interactive Effects of Testing #1 - Text

- the pretest may make the participant more aware of or sensitive to the upcoming treatment - as a result, the treatment is not as effective without the pretest - may be a problem in any design with a pretest - EX: suppose a fitness program is the experimental treatment; if the physical fitness test is administered to the sample first, the participants in the experimental group might realize that their levels of fitness are low and be particularly motivated to follow the prescribed program closely → in an untested population, however, the program might not be as effective because the participants might be unaware of their low levels of physical fitness

Reactive Effects of Experimental Arrangements #3 - Text

- treatments that are effective in constrained situations (ex: laboratories) may not be effective in less constrained settings (ex: more like the real world) - may not be generalizable to the real world - this a persistent problem in laboratory based research (ex: exercise physiology, biomechanics, motor behavior) - EX: in a study employing high speed cinematography, the skill to be filmed must be performed in a certain place, and joints must be marked for later analysis - a specific type of reactive behavior is the "Hawthorne Effect" → a phenomenon in which participants' performances change when attention is paid to them, which is likely to reduce the ability to generalize the results

Continuum of Experimental Research - Notes

- true laboratory control → eliminating extraneous variables → EX: animal model studies - quasi field realism → conducting experiments in the field; "real world" → less control but more realism *Experiments are done along this continuum - look at diagram on slide

Elimination - Notes

- use a homogenous (not different) group instead of a heterogenous (different) group → EX: age = 20 to 50 (heterogenous) → EX: age = 20 to 25 (homogenous)

Reversal Design - Text

- used increasingly in school and other naturalistic settings and is depicted as: O1 O2 T1 O3 O4 T2 O5 O6 - the purpose here (as with the time series) is to determine a baseline measurement (O1, O2), evaluate the treatment (change between O2 and O3), evaluate a no-treatment time period (O3 and O4), evaluate treatment again (O4 and O5), and evaluate a return to a no-treatment condition (O5 and O6) - this design is sometimes called: A-B-A-B-A (or sometimes just A-B), where A is the baseline condition and B is the treatment condition - statistical analyses for reversal designs also need to be regression tests of the slopes and intercepts of the lines between various observations

Threats to Internal Validity and External Validity - Notes

- variety of threats - controlling threats → randomization - failure to randomize when possible is an unrecoverable error - research design - selection of subjects, independent variables, and dependent variables *Read this section carefully in Chapter 18!

Necessary and Sufficient Conditions (3) - Text

- we may think of cause and effect in terms of necessary and sufficient conditions - EX: if the condition is necessary and sufficient to produce the effect, then it is the cause - however, alternative situations exist as well: (3) 1. Necessary but not sufficient = some related condition likely produces the effect. 2. Sufficient but not necessary = some alternative condition is likely the cause. 3. Neither necessary nor sufficient = some contributing condition is likely the cause.

Interaction of Selection Bias and the Experimental Treatment #2 - Text

- when a group is selected on some characteristic, the treatment may work only on groups possessing that characteristic - may prohibit the generalization of results to participants lacking the particular characteristics (bias) of the sample - EX: a drug education program might be effective in changing the attitudes of college freshmen towards drugs but this same program would probably lack effectiveness for third year medical students because they would be much more familiar with drugs and their appropriate uses

Multiple-Treatment Interference #4 - Text

- when participants receive more than one treatment, the effect of previous treatments may influence subsequent ones - a problem when the same participants are exposed to more than one level of treatment - EX: suppose participants are going to learn to move to the hitting position in volleyball using a lead step or a crossover step; we want to know which step gets the players in a good hitting position most quickly; if the players are taught both types of steps, learning one might interfere with (or enhance) learning the other, thus, the researcher's ability to generalize the findings may be confounded by the use of multiple treatments → a better design would be to have two groups and each group learns one of the techniques

The Inference of an Experiment - (2) Questions - Notes

1. Are the sample subjects representative of the population of individuals? 2. Are the variables of the study representative of the variables in the population? - *review diagram on slide - *the tighter the control, the farther from the "real world"

True Experimental Designs - Randomizations - Notes

1. Between Groups Factor - Randomly assign subjects to groups - Randomly assign groups to conditions 2. Within Subjects Factor - Randomly assign condition order - Counterbalance ("Latin Squares") condition order 3. Mixed Design -Both

Internal Validity - What makes it a good study? (5) - Notes

1. Control of extraneous variables 2. Manipulate independent variables 3. Measure dependent variables - reliability and validity 4. Correct Statistics 5. Appropriate and Effective Methods - *these rule out alternative hypotheses

CONtrol of Extraneous Variables - (5) Methods - Notes

1. Elimination 2. Assimilation 3. Matching 4. Statistical 5. Randomization

Cognitive Validity Problems: Effect Both Subject and Researcher (4) - Notes

1. Halo Effect - respond like you should (positively) - want to do well 2. Hawthorne Effect - anything new or special attention - EX: "new car feel" eventually goes away 3. Expectancy Effect - it should work 4. Avis Effect - control subjects (if they know they are control) try harder Controlling Problems: - placebo control → new and different → deals with Halo and Hawthorne - blind or double blind → participant no knowledge of experimental or control → participants and researchers no knowledge → deals with expectancy effect → deals with avis effect

Threats to Internal Validity (9) - Notes

1. History - effect of unexpected events 2. Maturation - age, growth, development, natural change 3. Testing - sensitivity - one test of "Y" affects a 2nd test of "Y" 4. Instruments - calibration, observer effects/observer inconsistency 5. Regression Effect - high to low, low to high - high on pre test but low on post test - low on pre test but high on post test - "regression to the mean (average)" 6. Selection Bias - non randomly assigned groups 7. Experimental mortality - subject dropout 8. Subject Selection-maturation interaction - time affects one group different than another 9. Expectancy - it (experiment) should work

Threats to Internal Validity (9) - Text

1. History 2. Maturation 3. Testing 4. Instrumentation 5. Statistical Regression 6. Selection Bias 7. Experimental Mortality 8. Selection-Maturation Interaction 9. Expectancy * Any of these 9 threats to internal validity may reduce or undermine the researcher's ability to claim that the manipulation of the independent variable produced the changes in the dependent variable.

Research Validity - (2) Measures - Notes

1. Internal Validity - the results can be attributed to the experiment 2. External Validity - the results can be generalized (inferred) to the population of interest in a realistic situation

Types of Designs (3) - Text

1. Pre experimental Designs 2. True Experimental Designs 3. Quasi-Experimental Designs

Experimental Design (3) - Notes

1. Preexperimental Design - no randomization - low internal validity and external validity 2. True Experimental Design - randomization - sufficient internal and external validity 3. Quasi-Experimental Design - some randomization - lower internal validity and higher external validity

Controlling Threats to Internal Validity (4) - Text

1. Randomization 2. Placebos 3. Blind Setups 4. Double-Blind

Types of True Experimental Designs (3) - Text

1. Randomized-Groups Design 2. Pretest-Posttest Randomized-Groups Design 3. Solomon Four-Group Design

Uncontrolled Threats to Internal Validity (3) - Text

1. Reactive or Interactive Effects 2. Instrumentation 3. Experimental Mortality

Threats to External Validity (4) - Text

1. Reactive or Interactive Effects of Testing 2. Interaction of Selection Bias and the Experimental Treatment 3. Reactive Effects of Experimental Arrangements 4. Multiple-Treatment Interference

Threats to External Validity (4) - Notes

1. Reactive or interactive effects of testing - pretest increases the treatment effect 2. Interaction of selection biases and experimental treatments - the treatment works on the unique group 3. Reactive effects of experimental arrangements - it works in the lab but does it work in the real world 4. Multiple treatment interference - within subjects repeated measures designs, proceeding treatments effect following treatments *review examples on slide

Types of Quasi-Experiment Designs (6) - Text

1. Reversal Design 2. Nonequivalent-Control-Group Design 3. Ex Post Facto Design 4. Switched-Replication Design 5. Time-Series Design 6. Single-Subject Design

Pre Experimental Designs (3) - Notes

1. T O 2. O1 T O2 3. T O1 ___________________________ O2

External Validity - Notes

1. for who, what, and/or where are the findings appropriate? 2. who is this meaningful for? - generalizability (transferability) → inference = study to the real world

(1) Non-Parallel lines mean? (2) Parallel lines mean?

1. non parallel = interaction 2. parallel = no interaction

Pre Experimental Design #2 - Notes

O1 T O2 - O1 < O2 - other factors than "T" explain the difference: → Historical → Maturation → Regression Effect → Testing sensitivity → Placebo, Halo, Hawthorne, Expectancy - better design than 1, but not acceptable

Experimental Design Symbols - Text

R = random assignment of participants to groups O = an observation or measurement T = treatment is applied "------" = goes between groups; the groups are used intact rather than being randomly formed Subscripts = indicate either the order of observations and treatments (when they appear on the same line) or observations of different groups or different treatments (when they appear on different lines) → EX: T1 and T1 appear on different lines, they refer to different treatments; when they appear on the same line, they mean that the treatment is administered more than once to the same group

True Experimental Design 3 - Factorial Design - more than 1 independent variable - Two Factor Design - A (3 levels) by B (2 levels) - Notes

R A1 O1 B1 R A2 O2 R A3 O3 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ R A1 O4 B2 R A2 O5 R A3 O6 * review slide(s)

True Experimental Design 4 - Mixed Design - Two Factor Design - 2 (T/C) by 2 (Pre/Post) - Notes

R O1 T O2 R O3 O4 *review slide

True Experimental Design 5 - Solomon Four Group - Notes

R O1 T O2 R O3 O4 R T O5 R O6 - powerful design - replication of T: (O2 > O4) and (O5 > O6) - change due to T: (O2 - O1) > (O4 - O3) - testing effect: (O4 > O6) - testing treatment interaction: (O2 > O5) *review slide

True Experimental Design 2 - Randomized Group Design - One Factor Design with 3 Levels - Notes

R T1 O1 R T2 O2 R T3 O3 - the independent variable or factor has 3 levels instead of 2 - expanding the basic acceptable design

True Experimental Design 1 - Randomized Group Design - One Factor Design with 2 Levels - Notes

R T1 O1 (Experimental Group) R T2 O2 (Control Group) - O1 > O2 → most threats to internal validity are satisfied (expectancy, Hawthorne, Halo) → placebo control

Pre Experimental Design #1 - Notes

T O - many things other than "T" could have caused "O" - "You were in a strong wind, T. You could catch the flu, O."

Pre Experimental Design #3 - Notes

T O1 ____________________ O2 O1 > O2 - other factors than "T" could explain the difference: → groups unequal to begin with → selection bias