Ch. 9: Significance Testing

What are the steps of hypothesis testing?

1. State hypothesis -Null & research 2. Set significance level 3. Compute statistics 4. Make decision -Reject or fail to reject null?

What is Type I error? What are the consequences of it?

Type I error: Rejecting null hypothesis that is actually true -concluding difference exists when it doesn't -likelihood of making this error is equivalent to level of significance (alpha) -on any one test of null hypothesis, there is 5% chance you'll make a Type I error Consequences: Bad for science and scientists -may wast time trying to replicate effect -will be discredited when study doesn't hold up to replication -slows down progress of science

What is the p-value?

The probability of the observed results if the null hypothesis is true -Any given outcome is associated with p-value p lower that 0.05 = reject null -there's less than a 5% chance that results we observed were due to chance -havent proven anything (still 5% chance we are wrong) p higher than 0.05 = accept null; fail to reject null -research hypothesis may still be right, but data doesn't provide strong enough evidence to reject null

What is the critical value?

the minimum value needed to reject null hypothesis Each test statistic has table of critical vales based on alpha level and sample size Tables tell us minimum vale needed to reject null i.e. Table tells us obtained value that corresponds to significance level (typically p<0.05) you've set as alpha

What are confidence intervals?

An estimated range for population value, given descriptive statistics from sample Uses Z-talbe and descriptive statistics from sample to calculate range associated with certain level of confidence Formula: x̅ +/- Z(SD) Remember it's two-tailed

What is the difference between significance and meaningfulness?

Statistical significance is used to back up argument that some variable matters But significance doesn't necessarily always equal meaningfulness ex: compare 5000 females to 5000 males 50% females have college degree 50.2% males have one =test may reveal statistically significant difference, but is difference meaningful? Statistical significance has to be considered in context: To be meaningful a study must be sound -Did it measure key constructs/variables appropriately? Is sample sufficient in size and makeup? Need for replication -Is this the only time effect has been observed? Absolute size of difference has to be considered -EFFECT SIZE p= 0.05 is relatively arbitrary -are results with p-value of 0.051 worthless?

What is the file drawer effect?

Studies that final to reject null are less likely to get published than studies that reject null Ex: 10 studies find men are better than women at math; 90 find they're not 10 that find significant difference are more likely to get published But, it may be that those 10 are type I errors Apparent consistency in published literature makes us think null is false

What is the alpha level (α)?

The accepted cut off for calling result "statistically significant" Degree of risk you're willing tot make to reject null hypothesis that is actually true Traditionally 0.05 or 5%

What is Type II error? What are the consequences of it?

Type II error: Failing to reject null that is actually false -concluding no difference exists when it really does Consequences: -May stop studying something important -Important research findings never make it to light -Truth goes undiscovered

What is statistical significance?

low probability of obtaining at least as extreme results given that the null hypothesis is true Hypothesis testing is process of determining statistical significance