Research Methods Exam 3

The independent variables

(IVs are called factors)

The independent variables

(IVs are called factors)With factorial designs there are many, at least two.

ABA Reversal Designs:

- A is a "baseline" phase, no treatment. - B is the treatment phase. • At a minimum: - No treatment baseline. » Target behavior is observed. - Treatment phase . » Manipulation is conducted. - Return to baseline phase. » Treatment is withdrawn - With the ABA version it ends in the withdrawal of the treatment.

• Interactions

- Behavior is determined by many things, rarely just a single factor. - Sometimes IVs have no effect in isolation. - But, when the IVs are combined there is an effect on behavior. - Interactions and their assessment is critical in factorial research. » This is why we conduct factorial research. » If we were not interested in how two variables combine to affect behavior then would just use a single variable design.

Section 2: Single Variable Correlated Groups & Single Subject Design

- Can we do this without RATG? - Yes, with correlated groups designs. It's an experiment because Using same participants in all conditions guarantees equivalency of the "groups" because it is the same person!

- Within Subjects or Repeated Measures Factorial Design:

- Each participant appears in only one cell. - There is RATG for each cell. Pre ATTA Post ATTA Left 35 67 51 Right 38 21 29.5 36.5 44 Hand Main Effect Factorial Designs • But, we could have used a completely within subjects design for the same study: - This would greatly increase our power. » Provide greater sensitivity to detect differences. » By reducing error from individual differences. - This would also ensure group equivalency at the outset of the study.

Characteristics of matched subjects designs

- Each participant exposed to only one level of the IV. - Each participant has a matched participant in each of the other conditions. -the groups are correlated - The analysis considers which participants are matched with with other participants. - Critical comparison is the difference between the correlated groups

Using Within Subjects Designs

- Each participant tested under each experimental condition. - Thus, scores in each condition are correlated with scores in other conditions. - Critical comparison is the difference between the conditions on the DV. • The problem is that the participants serve in all conditions. - They are exposed to all manipulations.

The effect of the IV on the DV demonstrated by:

- First, with the initiation of treatment the behavior changes in the predicted direction. - Then, with treatment is withdrawn there is a return to baseline. - This pattern continues as often as the "ABA" pattern continues. - This consistency in responding and not responding gives the study power. » And permits causal inferences

The effect on power

- Groups are equal at beginning since use same participants. - The removes the largest source of error variance. » That arising from individual differences. - Consider the F ratio: » Individual differences part of error removed. » The denominator thence decreases. » The F ratio increases. - Thus, the analyses are more sensitive to small differences. » Statistical power is increased. » Increased without having to increase sample size. • The results, in terms of F ratio and p value, interpreted the same way as before.

Randomization:

- Here the order of presentation of conditions is randomized. - Each participant then receives a different, random order of presentation. - Use a table of random numbers. - Thus, sequence effects are just as likely to occur in one condition as in another.

Between and within subjects factorial designs:

- Here there is at least one factor to which different subjects are assigned to the different levels - There is at least one other factor for which subjects are exposed to all conditions.

ABAB.

- If a true treatment initiated, would you want them to walk away untreated. » Particularly if an effect of the treatment was noted. » You would be doing them harm. - Hence, these designs are often ABAB.

What to match?

- Match variables that can have an effect on the DV but that are not of direct interest in your study. - The more potent the influence on the DV, the more important it is to match on that variable. Match on a couple or just a few of the most important variables. » Variables that are strongly related to performance on the DV.

- Analysis of Variance (ANOVA):

- Our IV may have more than two levels.- ANOVA can test for differences between a set of two or more means. - "oneway" ANOVA means that there is just one IV. » But, can have multiple levels

Counterbalancing:

- Systematic arrangement of the order of conditions so that all possible orders or positions of conditions are represented an equal number of times.

• The effect of any combination of IVs on the DV.

- There may be more than one interaction, depending on the design of the study. We are interested in whether the interaction exists. - Often, the results from main effects don't really tell us anything.

Multiple Comparisons: 2 types

- Use t-tests to find where significance lies. • Two classes of multiple comparisons: A priori and Post hoc

Multivariate Analysis of Variance (MANOVA):

- We have been discussing factorial designs. » We are using more than one IV. » Use the ANOVA to analyze. This test analyses all the dependent measures under the umbrella of one criterion level. » Again, though, will only tell you that one DV had an effect. MULTIPLE DV

types of effects: Carryover and Novelty

-Carryover effects are due to influence of a particular condition or a combination of conditions on later responses. » A particular condition may affect any condition that follows. » Consider psychophysiological research. » The anger study, inducing anger may affect later responses. - The novelty effect. » Again, with psychophysiological research. » First condition often generates bigger response due to novelty effects.

Two general types of counterbalancing:

>Complete >Partial

A priori

A priori are when we specified at the outset which comparisons we were interested in making. » We then make these comparisons after finding a significant F ratio.

ANOVAs all do the following:

Compare the variability between the groups to the variability within the groups. • Even with complicated designs this stays the same. - Subtle changes in computation of different variance components for between versus within subjects designs.

Main Effects and Interactions:

Different Hypotheses: • The effect of each IV on the DV. - These are the main effects The effect of one IV irrespective of the other IV. » Main effects go across the levels of other IV.

Matched Subjects Designs:

Does not use each participant as their own control by testing each participant under each condition . - Rather, different participants in each condition. - But they are closely matched before being assigned to the conditions.

Nonexperimental Approaches

Ex Post Facto Studies: • Observe present behavior and attempt to relate to prior experiences. - But, little confidence in validity due to lack of controls. - There is no manipulation. - Hence, we are not creating systematic variance. • As an example: - As clinicians we see patients and they often report experiencing sexual abuse in their past. - So, sexual abuse leads to psychopathology. - Remember the ex post facto fallacy? » Can we then conclude that sexual abuse leads to psychopathology? » May individual experience such abuse and never experience psycholological problems. We cannot control for possible confounds. - Hence alternative hypotheses cannot be ruled out.

Two sources for systematic effects:

Experimental variance - what we force into the system. » Extraneous variance - from uncontrolled variables, confounds. »(Together these are known as systematic between groups variance)

Latin Square Design: memorize this rule » 1, 2, N, 3, N - 1, 4, N - 2, 5, N - 3, 6, N - 4, 7, etc.

First, randomly order the conditions of the experiment. » Number of conditions is designated as N. » Size of the square will be N x N. » Number your random order 1 through N. » Anger (1) Sad (2) Happy (3) Fear (4) Joy (5) Funny (6) - Second, generate first row of your square. » Use the following rule. » 1, 2, N, 3, N - 1, 4, N - 2, 5, N - 3, 6, N - 4, 7, etc.

Negative practice effects

For negative practice effects include a rest period. » Allow the fatigue to dissipate between conditions. » May be especially important if testing all day, as we often do in neuropsychological research. - May also use equivalent, alternate forms as we do in neuropsychological research • These will minimize practice effects, but best control is to vary order of presentation.

- One factor may be between subjects.

Hence, different subjects are assigned to each level of that factor.

Multiple baseline designs:

Here the effects of treatment are demonstrated on difference behaviors in a successive fashion. » Each behavior is targeted. » But at different time points, successively.

Within Subjects Designs:

Here the participants are exposed to all experimental conditions. each person serves as his own control. • Since each person is in each condition, this design sometimes called a repeated measures design. - Everybody gets everything.

Multilevel Completely Randomized Between Subjects Design:

Here, participants randomly assigned to three or more conditions. - So, we examine several different emotions, not just two.

Matched subjects are improvements and potential problems

It ensures that the groups are correlated, equal, on important dimensions at the outset. » Hence, it is a sort of marriage between independent groups design and within subjects design. • Potential Problems: - What happens in the case of outliers? » You will have to exclude people for whom there is no match. » Hence, a type of attrition in a way. » How might that affect the conclusions that you draw, your inferences from the findings? - What if you have more that one condition? » The more conditions (groups) you have the harder it will be to find enough matches.

Remember, we want to:

Maximize experimental variance. - Control extraneous variance. - Minimize error variance

Strengths and weaknesses of within subjects designs:

No group differences due to sampling error. - This is because same participants are being used. - This guarantees group equivalency at the outset. - Hence, confounds due to initial group differences are eliminated. • More sensitive than between subjects designs. - We want to minimize error variance. » Which is due to individual differences, Within subjects designs eliminate this source of variance .Fewer participants are needed. • Increased efficiency. - May only need to give the instructions once.

• Partial counterbalancing.

One method is to randomly select some of the sequences of conditions and then randomly assign participants to those. - Another method is to use Latin Square Design.

Types of sequence effects:

Practice effects result from subjects gaining experience with the test, the procedure. » Caused by "practice" and increased familiarity. » Positive practice effect is when performance is enhanced. » Negative practice effect is when performance decreases, diminishes as may happen with fatigue. » Practice effects can occur regardless of the sequence used.

- Controls for sequence effects: Practice effect

Practice effects. - For positive practice effects hold the variable constant. Single Variable Correlated Groups and Single Subject Designs » Train participants to the same level. » All participants are then equally as skilled and familiar

Random Assignment used to control

Reduce extraneous variance by ensuring that the manipulation is the only difference. » Extraneous variance nicely controlled by RATG. » This will make the groups equitable » Make the variables constant - make participants homogenous. » But, this may limit generalizability. » Build the confound into the study as another IV. » » Match the participants or use a within subjects design(Run a tightly controlled study! » This increases our power and confidence.)

- Steps of analysis and interpretation:

Remember, alpha level set at .05. » We use this level for the main effects and for the interactions. » The entire ANOVA done under umbrella of .05. Factorial Designs - When using factorial designs we interpret the results for both the main effects and the interaction. » We evaluate whether they meet our .05 criterion. » BUT, always begin with interpretation of the interaction. » Only with a significant interaction do we have permission to evaluate differences between the cell means.

sequence effects

Sequence effects concern the influence of one condition on subsequent conditions.

Sampling error

Significant differences in group means indicates that variability is larger than would be expected due to chance. » Due to the natural variation that occurs when drawing samples from a population.

Single Group Posttest Only Studies

Somewhat higher level of constraint here. - Because there is manipulation of an IV. - But, there is no control group, no comparison. - There is also only one measurement taken after the manipulation. • The procedure? - Draw a sample. - Conduct some manipulation. - Then take a measurement. • Hence, many factors left uncontrolled, including: Placebo effect. - History. - Maturation. - Regression to the mean.

Sources and Forms of Variance

Systematic Between Groups Variance and Nonsystematic Within Groups Variance

Solomon's Four Group Design

The pretest may sensitize the subjects, it may affect later responses. » The pretest may be a type of "pretreatment." a combination of the randomized pretest posttest control group design and the posttest only control group design.

Multivariate Analysis of Covariance (MANCOVA):

This is a combination of the ANCOVA and the MANOVA. » Here we have multiple IVs. » We have multiple DVs. » And we have a known confound that we want to control through statistics. MULTIPLE DVS AND CONFOUND NEEDED TO REMOVE

Systematic Between Groups Variance

This is systematic or planned variance. - This is variance between the groups caused by manipulation of the IV. - The variance that we predicted would occur and are testing for. - We are looking for significant differences in the variance between groups. » Meaning that the variability between the groups is larger than expected on the basis of sampling error or chance. (must keep in mind that difference may be due to experimental variance or extraneous variance.) » Just indicates that systematic effects existed to create the difference.

Memorize the rule

Thus, our first row would be Anger (1), Sad (2), Funny (6), Happy (3), Joy (5), Fear (4). - Generate the second row by adding 1 to each number of the first row, with 1 added to N equaling 1. » Thus, our second row would be Sad (2), Happy (3), Anger (1), Fear (4), Funny (6), Joy (5). - Generate the third row by adding 1 to each number of the second row (N + 1 = 1). » Thus, we get Happy (3), Fear (4), Sad (2), Joy (5), Anger (1), and Funny (6). - Generate the remaining rows using the same rules. » Do so until N rows are completed. - Subjects then randomly assigned to rows of the square. » Each row must be used an equal number of times. Thus, the total number of subjects must be a multiple of N

example

Thus, with 3 different conditions you will need 3 x 2 x 1 = 6. » As an example: ABC BCA ACB CAB BAC CBA » But with 4 conditions you need 24 sequences.

Analysis of Variance in Factorial Designs:

We use an ANOVA. • Results of an ANOVA presented in a summary table. - Summary table lists the sources of variance. - But with factorial designs there are more sources of variance.

Maximizing Experimental Variance

We want to be sure the IV really varied. » To do so use manipulation check. » Manipulation check ensures that our manipulation created a difference, had it's intended affect.

Controlling Variance in Research

We want to maximize experimental variance, control extraneous variance, and minimize error variance

Experimental Approaches: (Use RATG)

What is different here? • Inclusion of a control group. • RATG. • The combination of these controls many types of confounds. - Control group helps control against: » History. » Maturation. » Regression to the mean. » Placebo effect

When to Use Matched Subjects Designs:

When we want the advantages of correlated groups design. - But cannot use a within subjects design. » Such as when exposure to one condition causes long term changes in the participants, » Hence, any other subsequent condition would be affected and counterbalancing or randomizing would be useless.

Mixed Designs:

With more than one factor, they may be of different types. May refer to one or more factors being within subjects and one or more being between subjects. - May also refer to one or more factors being manipulated and one or more being non-manipulated.

pretest posttest

With pretest posttest design there is no counterbalancing. » The pretest must come before the treatment and the posttest after the treatment.

What goes into the ANOVA?

Within groups variance. » Measure of nonsystematic variation within a group. » Error or chance variation. » Average variability within the group. - Between groups variance. » Represents how variable the groups means are. - Sum of Squares. » Each source of variance has a sum of squares. Single Variable, Independent Groups Designs » Remember, this is the sum of squared deviations from the mean. » This is how the variances are calculated. • How is the ANOVA calculated? - Sum of squares for each source of variance calculated. » But, as with variance, is not scale specific due to squaring. - Hence, the sum of squares is divided by the degrees of freedom (df). » The result is the mean square (MS). - The MS for the Between Groups is divided by the MS for the Within Groups. » The result is the F ratio.

within subjects

Within subjects designs use randomization or counterbalancing to control for sequence effects. » It is the use of randomization or counterbalancing that makes within subjects designs true experiments.

Systematic between groups variance is also:

a combination of both systematic between groups variance (experimental and extraneous) and nonsystematic variance due to sampling error.

Nonsystematic Within Groups Variance:

aka error variance. it's due to random factors that affect participants differentially within the same group. » Systematic reflects variance among all subjects, across groups. - Nobody is the same,

- Single Group Pretest-Posttest Studies:

an improvement on the previous type of study. - Here there is now a pretest taken prior to the manipulation. - We can now assess, or verify, that a real change occurred. - But, the same factors are still uncontrolled.

Section 3: Factorial Designs

interactions vs main effects/ boring because=broad generalization/ average across other variables, so they lose meaning

Randomized Posttest Only Control Group Design

most basic level. - Includes randomization and control group. - Subjects are randomly assigned to either a control group or a treatment group. - Then a manipulation or treatment occurs. - A measurement is then taken.

bonferroni= most conservative Tukey Least Significant difference=least conservative

must be mindful of the Type I error rate. - Need to control for experiment wise error rate. - Each comparison we add .05. - Thus, our chances of making Type I error increases. - Need to control for this. » Tukey Honestly Significant Difference » Tukey Least Significant Difference » Newman-Keuls » Sheffe » Bonferroni

to make causal inferences

must show that experimental variance is due to manipulation of the IV. - Experimental variance must be high, and not washed out or distorted from too much extraneous variance or error variance. » The more extraneous and/or error variance you have the more difficult it is to show the effects of systematic experimental variance

Post Hoc

when we did not specify which comparisons were of interest before the study. » We just go in and make a bunch of comparisons.

Hypothesis Testing:

so many different hypotheses, there are more chances for confounding to occur. - Factorial designs are complex. - So may be the threats to validity. » We must rule out alternative explanations for each factor involved. » Each factor must have controls associated with it. » Random assignment for each cell is good protection. • Interpretation of interactions is also much more complex. - We must consider that the effects of one IV depend on the level of the other IV.

Minimizing Error Variance:

sources» Measurement error - variations in the way participants respond, may come from unreliability of the instruments, for instance. » Individual differences - remember the snow flake idea? - What do to? » Maintain carefully controlled study, controlled and reliable measurements. » Individual differences controlled through randomization, whenever possible randomize. » Also controlled through repeated measures design,

Analysis of Covariance (ANCOVA):

the partial correlation? » We wanted to assess the strength and direction of a relationship between two variables. » While controlling for the effects of a third variable. - ANCOVA also seeks to control for effects of a third variable. » CONFOUND YOU KNOW ABOUT AND CONTROL

Randomized Pretest-Posttest Control Group Design

the same as the pretest-posttest natural control group design. - The difference? - RATG. What to compare? - The posttest measurements between the two groups. - What about the pretest? » With the pretest we can be assured that the groups were equal at the outset. » Give us greater confidence. • The posttest comparison is actually critical

Extraneous variance

uncontrolled, the effect that these variables have on the results.

Section 1: Single Variable Independent Groups Design

variance and planning

- Characteristics of complete counterbalancing:

» Each participant exposed to each condition. » Each condition presented an equal number of times. » Each condition presented an equal number of times in each position. » Each condition precedes and follows each other condition an equal number of times.

Controlling Extraneous Variance

» Experimental and control groups need to be as similar as possible at the outset. » Groups are treated exactly the same (except for the manipulation, of course). - The IV manipulation must be the only difference!

- One factor may be within subjects

» Hence, every subjects gets exposed to every level of that factor

With between subjects:

» If we want 20 per cell then would need 80 subjects. » But, for within subjects design would only need 20 subjects. » Thus, need fewer participants. - May also be more efficient. » This may particularly be the case with factorial designs in which there are multiple IVs.

This is the only control for carryover effects. - Two methods to do this:

» Randomization » Counterbalancing The idea is to control sequence effects by spreading out these effects equally across all conditions.

What if there are several such variables?

» Then go with the ones that have the greatest variability in the population. » Why? » Because with greater variability there is an increased chance of extraneous variance affecting the study. » There is a greater chance for differences between the means to occur by chance with random assignment.

Multiple Comparisons:

» With more comparisons come increased risk of Type I error. Factorial Designs - We can use the same correction for experiment wise error rate: » Tukey HSD » Tukey LSD » Scheffe » Newman-Keuls » Bonferroni

examples of design notation, main effect, and interaction

• Design Notation - 2 x 3 - 2 (Sex: Men and Women) x 3 (Location: Home, Car, Office) • Main Effects - For Sex » 5.67 versus 7.00 - For Location » 5.50, 6.50, and 7.00 • Interaction - Within the individual cell means.

What does it all mean?

• F-ratios are calculated for all the main effects and interactions. - Between variance/within variance. • Obtained probability is compared to our criterion for each, i.e. the alpha level (p < .05). - Nothing changes here. - We have null hypotheses for all the effects. - We reject if obtained probability falls below the criterion. • The key is in the interpretation phase. - Here we have to interpret and explain. » All the main effects. » But also all the interactions

Design Notation:

• Shows how many IVs and how many levels to each IV. • Thus 2 x 2 indicates: - Two IVs. - Two levels to each IV. • A 2 x 4 indicates: - Two IVs. - One with two levels and one with four levels. • A 2 x 3 x 3 indicates: - Three IVs. - One with two levels, one with three levels, and another with three levels.

Possible Outcomes for Factorial Designs:

• There may be main effect for one factor, or more than one. • May be no main effects, no interaction either. • May be an interaction between two factors. • May be an interaction between more than two factors. - A graph of the data will indicate an interaction. • With an interaction the lines are not parallel. 0 20 40 60 80 Left Right Hand ATTA CI Pre ATTA Post ATTA Factori

Pretest-Posttest Natural Control Group Studies

• Yet even higher constraint here. - Now, we have added a no-treatment control group. - A group that does not receive the manipulation. - But, the control group is naturally occurring. Hence, there is no RATG. The problem? - We cannot know whether the groups are equal at the outset. » We could test on some variables to determine equality. » But, we cannot possibly know all of the potentially confounding variables.