Effect size and Power

what is a small effect size according to cohens d

0.2

what is a medium effect size according to cohens d

0.5

therefore, what do we set the power value to

0.8

what is a large effect size according to cohens d

0.8

How does sample size affect power?

The larger the sample size, the greater precision we have in the estimation of the population parameters. In general, as the sample size increases, so does the likelihood of detecting significant effects (such as differences between group means), hence the power of the test increases.

what is the alpha value

probability of making a type 1 error (false positive) that a researcher is willing to make (usually it is 0.05) if the p value is less than alpha, then you can reject null w/ confidence

what is power

refers to the power we would like to have in the test to be able to detect a significant effect should one exist in the population. It is calculated by: 1 - b

what are unbiased effect sizes

take into account sample size to provide a 'truer' measure of the effect size in the population. Commonly used unbiased effect sizes include: Hedges' g: •Usually used for t tests (Equivalent to Cohen's d) Omega squared Usually used for ANOVA (Equivalent to eta squared) Adjusted R squared Usually used for Regression (we have come across this one already)

what would happen if you changed the alpha value to 0.01

the power of the test will decrease. With a lower alpha, it becomes more difficult to detect significant effects.

what would happen if you changed the alpha value to 0.1

the power of the test would increase

What is the beta value?

the probability of making a type 2 error. Type 2 error = Retaining the null hypothesis even though there is a genuine effect in the population. The beta value is not as 'set in stone' as the alpha value, but generally we want to set the beta value to 0.20 this is more than the alpha value because making a type 2 error is generally considered less harmful than making a type 1 error.

what is the issue with all of the above measures

they are referred to as biased estimates. Essentially this means that they tend to overestimate the 'true effect size' in the population. This is a particular problem when the sample size is small.

what are we looking for in power analysis

whether the actual power of a test reaches 0.80

what do we use to do a power analysis

GPower

effect size for correlation

In addition to r being used to calculate the p value, Pearson's correlation r , is often used as an effect size. Small effect size r = .10 Medium effect size r = .30 Large effect size r = .50

what is effect size

a measure of the strength of the relationship between two variables. Given a large enough sample size, virtually any effect can become significant, but is it meaningful in real world terms

Setting alpha at 0.05 and beta at 0.20 (and therefore power = 0.80) strikes a good balance:

It minimises the amount of both types of errors, without requiring a much larger sample size which we would need if we set both alpha and beta at lower values.

The effect size for ANOVA in GPower is:

Small effect size f = 0.10 (difficult to detect) Medium effect size f = 0.25 (still not easy to detect) Large effect size f = 0.40 (easy to detect)

what is the power of a statistical test

The probability that a statistical test will yield a statistically significant result when not significant

The power of any particular statistical test depends on 3 main parameters:

The value of alpha, or significance level (almost always set at 0.05) The effect size. The larger the effect, the more likely it is that a genuine effect will be detected. The sample size. The larger the sample size, the more likely it is that a genuine effect will be detected.

Why is this important?

Using these concepts allows us to do two very useful things: -We can look at whether a non significant result for any statistical test is likely to be due to a lack of power in the test. AND -We can use these concepts to calculate what effect size we are likely to need before conducting a full study, rather than just relying on fairly arbitrary criteria such as 'at least 30 cases'.

why is power analysis for factorial anova more difficult

as we have to calculate power for each of the main effects separately, as well as for the interaction effect(s)

effect size for ANOVA

eta squared Interpreting η2 (and partial η2) as a rule of thumb (Cohen): .01 = small .06 = medium .14 = large

we can estimate the effect size we expect to find: the process of running a power analysis a priori

• Base it on substantive knowledge •(e.g. I'm looking for an effect, a change in mean score on an exam, say, of 5%. If I don't get this, then the results are not that meaningful anyway). • Base it on previous research •(e.g. Look at similar studies & see what effect size others have found) • Use conventions •( We could assume a medium effect size if have no other info.) • Run a pilot study •(If you have a sample of about 30, then this is usually sufficient to give you a reliable estimate of your effect size)