POST HOC COMPARISONS & TESTING ASSUMPTIONS

Assumptions of the F-ratio

* Independence - The numerator & denominator of F-ratio are independent * Random Sampling - Observations are random samples from the populations * Homogeneity of Variance - The different treatment conditions have the same variance * Normality - Observations are drawn from normally distributed populations

Post Hoc Analytical Comparisons

A variety of different post hoc tests are used e.g. * Scheffe * Tukey HSD * T-tests These tests vary in their ability to protect against Type I errors * Increasing Type I protection reduces Type II protection

Testing Assumptions of ANOVA

Each of these assumptions should be met before progressing onto analysis. 2 assumptions that we have to assume have been met by the experimenter: * Independence and random sampling If an experiment has been designed appropriately both of these assumptions will be true However homogeneity of variance and normality assumptions need not necessarily be true

Testing Homogeneity of Variance- A Heuristic

For hand calculations, there is a quick and dirty measure of homogeneity of variance: largest variance / smallest variance is < 4 Note: this is a heuristic. When you have the option, use one of the specific tests e.g. Bartlett

Per Comparison and Familywise Error

If we perform several statistical tests on a set of data we can effectively increase the risk of making a Type I error. * If we perform 2 statistical tests on the same set of data then we have a range of opportunities of making a Type ! error: Type I error on the first test, Type I error on the second test, Type I error on both the 1st and 2nd test Type 1 errors involving single tests are known as per comparison errors The whole set of Type I errors above is known as the familywise error

Data Transformations

In order to return our data to normality and establish homogeneity of variance we can use transformations - These are simply mathematical operations that are applied to the data before we conduct an ANOVA - However, there are 3 circumstances where no transformation of the data will work: * Variance are heterogeneous * Distributions are heterogeneous * Variance and Distributions are heterogeneous Moderate skew- use square root. Substantial skew- use logarithm Severe skew- use reciprocal

Post Hoc Tests

Post hoc tests are conservative - they reduce the chance of Type I errors by GREATLY increasing the chance of Type II errors. - Only very robust effects will be significant - Null results using these tests aren't easy to interpret - Many different post hoc tests exist, have different merits and problems - Many post hoc tests are available on computer based statistical packages e.g SPSS, Experstat

Scheffe Test

Scheffe is calculated in exactly the same way as planned comparison - Scheffe differs in terms of the F critical that is adopted - For the one-way between groups analysis of variance, the critical F is associated with an F scheffe is given by: (SEE WRITTEN NOTES FOR EQUATION)

Testing Normality be Examining Skew

Since we assume that - the distributions from which the samples are taken are normal - and the skew of a normal distribution is equal to zero Then - One test of normality is to see if the skew is significantly different to zero - In other words, test the value of skew to see if it deviates significantly from a normal distribution

Testing Normality

The 3 most commonly used tests for normality are: - Skew - Lilliefors - Shapiro-Wilks These tests compare the distribution of data to a theoretically derived normal distribution - All of these tests are v. sensitive to departures from normality when there are large samples. - Lilliefors and Shapiro-Wilks hard to calculate by hand, but both available on SPSS

Tukey HSD

The Tukey (Honestly Significant Difference) test establishes a value for the smallest possible significant difference between two means. * Any mean difference greater than the critical difference is significant.

Per Comparison and Familywise Error Rates

The relationship between the 2 error rates is simple: a fw = c(a pc ) where c is the number of comparisons so...if we make 3 comparisons: 3*(0.05)=0.15 errors 20 comparisons 20*(0.05) we'll make on avg 1 error If we make 20 comparisons, it's possible we may be making 0, 1 or 2 or in rare cases even more errors * The chance we'll make at lease 1 error is given by the formula: 1 - (1-a)c

Transforming Data

Transforming data reduces the probability of making a Type II error - If an assumption is broken, ANOVA fails gracefully: we will miss real effects (Type II) but we will not increase our rate of making claiming effects that do not exist (Type I) * Data should be transformed when data is either not homogeneous or not normal - Solving the homogeneity problem often solves the normality problem & vice versa

T-Tests

When comparing 2 means, a modified form of the t-test is available. For multiple comparisons the critical value of t is found using p=0.05/c This is known as a Bonferroni correction

Testing Homogeneity of Variance

When looking at betweens groups design use: - Hartley's f-max - Barlett - Cochran's C When looking at within or mixed design use - Box's M All these tests are sensitive to departures from normality, and available in SPSS

Transforming Data (2)

When transforming the data is impossible... - In general we proceed with analysis but but advise caution to the reader when reporting the results - This is particularly important if the observed F value has an associated p value, such that: 0.1 < p < 0.01 In these circumstances it's difficult to know whether a Type I or Type II error is being made, or no error at all

Type I Error Rates and Analytical Comparisons

With planned comparisons: - Ignore the theoretical increase in familywise type I error rates and reject the null hypothesis at the usual per comparison level - With post hoc /unplanned comparisons between the means we can't afford to ignore the increase in familywise error rate.