Variance in Research Designs

Slide

#11

Slide

#12

Recommendations

, Remember that there are multiple ways to provide evidence for the hypothesis: 1. Confidence interval, effect size 2. NHST - most commonly used by tendencies are changing , Use the simplest analysis that makes your point , Include descriptive statistics and effect size , Understand limitations of NHST and the claims that can be made with this approach

Repeated Measures ANOVA

, Similar to independent groups with important differences , Partitioning the variance Sum of Squares , Total = IVa + (Subjects + Error (residual)) , For repeated measures design: A. calculate summary score for each Subject in each condition B. then summarize performance for each condition across all Subs , Variation due to individual subjects within a condition is eliminated when calculating summary score , Variation due to individual subjects across conditions is estimated as source of variance (Subjects) , Estimate of error variation: Residual error

Significance

, Statistical significance is not the same as practical or clinical significance ,These depend on: 1. External validity of the results 2. Practical consideration regarding cost and ease with which a treatment can be implemented with a larger group , Interesting story about malaria vaccine recently: 1. 40% reduction in severe cases of M (significant improvement) 2. high cost, short life, requires injection - practical significance?

Significance

, Statistical significance is not the same as scientific significance , Scientific significance depends on: 1. nature of variables under study 2. Internal validity of the study a. , Statistically significant result can be found in an experiment with confounds 3. Other criteria, such as effect size and variance accounted for

Anova Example

- 1 way independent group (between subject) ANOVA - Partitioning the SS (Sum of Squares) - Total (A: Due to IVa error + B: Due to error (chance)) - Degree of freedom - Source [IVa (a-1) + Error (a(n-1))] = Total (an-1) - a: # of levels for IVa - n: # of scores per level - Always compute your df to confirm with output!

2 way 2 by 2 Independent groups design

- 2 variables, each variable has 2 levels - Anova: is ideal for the NHST of Factorial design A. can test main effects and interaction together - There is an interaction effect is the differences are not equal

2 way 2 by 3 Independent groups design

- Anova: ideal for NHST of Factorial designs - 2 way 2 by 3 - 2 variables, one variable at 2 levels, one at 3 -

Sources of Variance

- As an experimenter you want to maximize primary variance, minimize error, and control secondary variance. - Can not completely eliminate error variance - minimize! - Effect = Primary / Error - Treatment / Chance - Secondary variance has the potential to impact both internal & external validity

Analysis of Variance

- As the amount of systematic variation increases (due to effect of IV), the expected value of the F ratio becomes greater than 1.0 - Use NHST to decide how much greater than 1.0 the F-test must be to reach statistical significance - This will depend on the degrees of freedom

Type Errors of NHST

- Because decisions about the outcome are based on probabilities, errors can occur - Type I error: H0 is rejected when it really is true (no difference) - probability of Type I = alpha (α), level of significance - Type II error: H0 is false (is a difference) but it is not rejected - probability of Type II = 1-alpha (= β)

Controlling Noise

- Chances of detecting a real effect of your IV will improve if you can reduce variability caused by secondary variables - Two ways to reduce noise by controlling 2nd variables - Isolation: conduct the research in a "controlled" environment. Allows control over many environmental variables - Holding constant: hold some secondary variables (individual differences) constant to reduce variability due to the variables - If successful these two forms of control will minimize noise that might be masking the effect of your IV - Successfully controlling noise will increase the internal validity by eliminating potential confounds - However, it may also reduce external validity (your ability to generalize the results of your research and make statements about other groups, settings, and conditions). - Researcher must find the balance between controlling 2nd variables & allowing 2nd variables to occur randomly - influence the degree of internal and external validity

Data Analysis and Interpretation

- Data Analysis Plan - continued - 1 way ANOVA examples - Types of Significance

Data Analysis and Interpretation

- Data Analysis plan - continued - 1 way Independent groups ANOVA - 1 way repeated measured ANOVA - Type of significance

Analysis of Variance : F test

- F test - determines whether variation due to IV is larger than what would be expected based on error variation alone - F = variation between groups / variation within groups - F = systematic variation + error variation/ error variation

The Logic of Anova - Error variance

- Identify sources of variance in the data - Primary variance (effect of IV) - Error variance - In properly conducted random groups design, the only difference within each group should be error variance - a result of chance happenings during data collection - Properly conducted = ? - Random assignment - Manipulation of independent variable - Control of confounding variables

Analysis of Variance : True null hypothesis

- If Null hypothesis is no systematic variation between groups (No effect of IV) - If there is no systematic variation (= 0), F is equal to 1 - The between group variation is not greater than what you would expect by chance (error variation)

Threats to Internal validity - Selection

- Independent groups (between subject) design threats - Occurs when participants in one group differ initially from participants in another group due to selection differences. Often the result of "intact groups" or due to lack/failure of random assignment of subjects to groups. - Example #1: the effects of watching violent TV programs on future aggressive behaviors. - Children come to the experiment having experienced more or less violent TV. Natural groups, no random assignment.

Threats to Internal validity : Mortality/ Attrition

- Independent groups (between subject) design threats - Occurs when you lose subjects from one level of the IV more than from some other level. Subjects may drop out of experiment or simply stop behaving correctly or as expected. - Example #2: one group receives a reward for responding while the other group get negative reinforcement - Will likely lose more subjects at one level of IV (one group) relative to the other level of IV (other group).

ANOVA - What is it?

- Inferential statistics test - using NHST, the Null hypothesis there is no difference - Determines whether IV has statistically significant effect on DV

Analysis of Variance (ANOVA)

- Most frequently used statistical procedure with 2+ conditions - Single factor (IV) experiment with 3 or more levels - Factorial design with 2 or more Independent Variables

Analysis of Variance : Steps of NHST

- NHST with ANOVA - Step 1: Assume H0- no effect of IV - Step 2: Compute F test and obtain p value - Step 3: Compare p value to level of significance (p < .05) - If F-test is statistically significant (p < .05) - Reject H0and conclude that IV had an effect on DV - There is a difference somewhere among the means - May need to run post-hoc test to determine where diffs are If F-test is not statistically significant (p > .05) - Do not reject H0no evidence for a difference

Stage 3 : Confirm what the data reveal

- Null Hypothesis Significance Testing (NHST) - Most common approach for confirmatory data analysis - Need to be cautious when using significance testing - Goal of NHST - Determine whether differences among conditions are greater than differences expected by chance (error variation)

Statistical Power

- Power of statistical test - likelihood a statistical test will allow researchers to correctly reject Null hypothesis - 3 factors affect power - Level of significance (alpha) - Size of effect for IV on DV - Sample size - Increase sample size or change manipulation to boost Power - If possible, you should conduct a power analysis in advance to determine how many subjects are needed for good test

Analysis of variance - Primary variance

- Primary variance (systematic variation) - source of variation between groups - due to effect of IV - If null hypothesis is true - then no effect of IV -> no difference between groups - any difference between groups can be attributed to error - If the null hypothesis is false (IV had no effect) - Means and variance for experimental conditions should differ - Differences should be systematic - due to IV - Differences between groups can be attributed to effect of IV

Threats to Internal validity : History

- Repeated Measures (within subject) design threats - History: the occurrence of an EVENT, other than the IV that produces changes in participants' behavior. - Event is not under the researcher's control - Longer interval between measurements means a higher likelihood that some event will alter participant's behavior - Example #4: The effects of a course on critical thinking on student's critical thinking. Measure ability at beginning and end of semester - check for difference/improvement - Other events (courses, websites) may be confounds

Threats to Internal validity : Maturation

- Repeated Measures (within subject) design threats - Maturation: changes that occur to your participants between measurements (older, wiser, stronger, etc.) - These changes are not related to your IV. Practice, boredom, and fatigue are considered maturational threats to internal validity ...even though these are short-term changes. - Example #5: effects of a preschool reading program on adjustment to a school environment

Threats to Internal validity : Regression

- Repeated Measures (within subject) design threats - Regression toward the mean (statistical regression): extreme scores will become less extreme with repeated measurements. - Low scores will regress up toward the mean and high scores will regress down toward the mean - Example #3: effects of a remedial math course for first year students on grades in college-level math courses. - Improvements may reflect course or regression to the mean

Threats to Internal validity : Summary

- Selection (groups start out different) - Mortality/Attrition (groups end up different) - Regression toward the mean (lows & highs go away) - History (events influence performance across time points) - Maturation (changes in participant alter performance) - can be short- or long-term changes

Experimental Sensitivity

- Sensitivity of experiment: likelihood an experiment will detect the effect of an IV when, in fact, the IV had an effect (Null Hypothesis is false). Sensitivity increases with good research design and methods - For example: Reduce noise, Minimize error variance, Repeated measure design

Null Hypothesis Significance Testing

- Step 1: Assume the conditions do not differ (H0) - Assume the independent variable did not have an effect on DV - Null hypothesis - there is no difference - H0 - Alternative hypothesis - there is a difference - H1 - Step 2: Compute appropriate statistic to test for condition differences (e.g., t-test, ANOVA, X2, etc.) - Obtain the probability value for statistic (p value) - If "statistically significant" - outcome has small likelihood of occurring under H0 (p< .05) - Reject H0and conclude that IV had an effect on DV - Diff between conditions is larger than what would be expected if error variation (chance) caused the outcome 4

ANOVA example

- Summary table: provide

ANOVA example

- Summary table: provide statistics about sources of variation in data -> partitioning variance into sources - Mean square: Sum of Squares/df - Mean square error: Sum of Squares/ df for error - F-test: Mean square for group/Mean square error - If the df is not correct, then the F is not correct

Three Stages of Data Analysis

1. Get to know the data - Inspect data carefully, identify errors in the data, consider whether the data make sense 2. Summarize the data - Use descriptive statistics (central tendency, variability) and graphical displays of data 3. Confirm what the data reveal - Representative of the population mean? - Perform inferential statistics analysis - Decide whether the data support hypothesis, or not - Attempting to build the strongest case possible

Sources of variance : Primary and Error variance

1. Primary variance - the variability in the DV that occurs as a result of the influence of the IV 2. Error variance - unexplained variance. Variability due to true chance happenings such as moment-to-moment fluctuations in your subject's performance or fluctuations in your ability to accurately measure your DV due to chance variations in accuracy of equipment. * P/E = 1 " No effect of IV on DV * P/E < 1 " No effect of IV on DV * P/E >1 " Some effect of IV on DV (Significant?) * Ratio between these sources is KEY to results 3. Secondary Variance - variance in the DV that occurs as a result of the influence of secondary variables. -Impacts on Internal Validity - Secondary variables can create "noise" in the data that make it harder to detect an effect of your IV. - Increase in overall variability in your DV can mask any effects of the IV (remember P/E) - If the secondary variable co-varies with the IV then the secondary variable would be considered a confound and represents a threat to internal validity - E.g., groups of children exposure to clowns before experiment (one group more likely to see rodeo, perhaps )

Confirm What the data reveal

1. Representative of the population mean? 2. Perform inferential statistics tests 3. Decide whether the data support hypothesis, or not Statistic = Primary variance (IV) / Error variation

Variance in Research in Designs

1. Sources of variance 2. Analysis of variance 3. Noise in experiments 4. Threats to Internal validity

Slides

10

Repeated measures ANOVA

5 subjects, 6 measured at each of 4 intervals , Descriptive statistics for each condition -> Mean (SD) , Design: 1 - way repeated measures with 4 levels Null hypothesis: at each level, there is no significant difference

Null Hypothesis Significance Testing

Because decisions about the outcome are based on probabilities, errors can occur - Type 1 error: Null hypothesis is rejected when it really is true (no difference) ; Probability of Type 1 = alpha, level of significance - Type 2 error: Null hypothesis is false (is a difference) but it is not rejected ; probability of Type II = 1 - alpha - Researchers are always tentative about conclusions - Finding "support" the H1, do not prove it [ replicate -> reliable ] - Errors (Type 1 and 2) can occur and other hypotheses may exist - Failure to reject does not = accept - other factors may have impact

Writing an F statement

F (group df, error df) = F Ratio, p = p value

Conclusions based on Results

If observed p value < .05 (for example) 1. Reject null hypothesis (there is a difference) 2. Conclude that IV had a statistically significant effect on DV 3. Calculate the effect size (if you didn't already) Effect Size : eta squared = SS effect/ SSeffect + SSerror Effect size: indicates the proportion of variance account for by IV Indicates proportion of variance accounted for by IV

What does the F statement tell us?

Statistically significant at p < .05, what is it telling us? 1. IV had an effect on Dependent Variable - Groups differ in performance (# recalled) 2. Group means differ but F-test does NOT tell us which of the means differ (for the four conditions) 3. Examine the means to interpret effect of IV (where diff exist) 4. Use descriptive stats to judge differences between means 5. Collapse across some group means (e.g, control vs others) 6. Run post-hoc comparison of means - Tukey, for example

Null hypothesis significance testing

What do we conclude when a finding is not statistically significant? 1. Do not reject the null hypothesis of no difference 2. Do not accept the null hypothesis A. Do not conclude that IV did not produce an effect 3. Cannot draw a conclusion about the effect of an IV A. Some factor in experiment may have prevented us from observing an effect of the IV (2nd degree variable? B. Most common factor : Not enough participants Calculate the effect size size to see how much variance is accounted for by IV. Important for future considerations

2 way independent groups

total = IVa + IVb + Due to error + Interaction IVa = a -1 IVb = b - 1 Interaction = (a-1)(b-1) Error = ab(n-1)