Chapter 9
meaning of statistically significant
"Significant" in the statistical sense does not necessarily mean "important." It means simply "not likely to happen just by chance."
the two circumstances to determine how small a P-value is convincing evidence against the null hypothesis
1) How plausible is Ho?: If Ho represents an assumption that the people you must convince have believed for years, strong evidence (small P-value) will be needed to persuade them. 2) What are the consequences of rejecting Ho?: If rejecting Ho in favor of Ha means making an expensive change of some kind, you need strong evidence that the change will be beneficial.
influences on how large a sample is needed
1) If you insist on a smaller significance level (such as 1% rather than 5%), you have to take a larger sample. A smaller significance level requires stronger evidence to reject the null hypothesis. 2) If you insist on higher power (such as 99% rather than 90%), you will need a larger sample. Higher power gives a better chance of detecting a difference when it is really there. 3) At any significance level and desired power, detecting a small difference requires a larger sample than detecting a large difference.
factors that increase the POWER of a test
1) Increasing the sample size (reduces the probability of a Type II error) 2) Increasing the significance level (increases the risk of a Type I error)
questions we must answer to decide how many observations we need
1) Significance level: How much protection do we want against a Type I error--getting a significant result from our sample when Ho is actually true? By using alpha = 0.05, there is a 5% chance of making a Type I error. 2) Practical importance: How large a difference between the hypothesized parameter and the actual parameter value is important in practice? 3) Power: How confident do we want to be that our study will select a difference of the size we think is important?
one-sample z test for a proportion
Choose an SRS of size n from a large population that contains an unknown proportion p of successes. To test the hypothesis Ho: p = po, compute the z statistic. Find the P-value by calculating the probability of getting a z statistic this large or larger in the direction specified by the alternative hypothesis Ha.
one-sample t test
Choose an SRS of size n from a large population with unknown mean. To test the hypothesis Ho: μ = μo, compute the one-sample t statistic. Find the P-value by calculating the probability of getting a t statistic this large or larger in the direction specified by the alternative hypothesis Ha in a t distribution with df = n - 1. Use this test only when (1) the population distribution is Normal or the sample is large ( n >/= 30), and (2) the population is at least 10 times as large as the sample.
statistically significant
If the P-value is smaller than alpha, we say that the data are statistically significant at level alpha. In that case, we reject the null hypothesis Ho and conclude that there is convincing evidence in favor of the alternative hypothesis Ha.
test statistic
Measures how far a sample statistic diverges from what we would expect if the null hypothesis Ho were true, in standardized units. That is, test statistic = (statistic - parameter) / standard deviation of statistic
For significance tests, "robust" means that the _______ is pretty accurate.
P-value
conditions for one-sample t test
Random: The data were produced by random sampling or a randomized experiment. Normal: The population distribution is Normal (based on graph) OR the sample size is large (n >/= 30). Independent: Individual observations are independent. When sampling without replacement, check that the population is at least 10 times as large as the sample.
significance tests: a four-step process
State: What hypotheses do you want to test, and at what significance level? Define any parameters. Plan: Choose the appropriate inference method. Check conditions. Do: If the conditions are met, perform calculations. Compute the test statistic. Find the P-value. Conclude: Interpret the results of your test in the context of the problem.
beware of multiple analysis
Statistical significance ought to mean that you have found a difference that you were looking for. In other settings, significance may have little meaning.
null hypothesis (Ho)
The claim we seek evidence AGAINST that is tested by a statistical test. The test is designed to assess the strength of the evidence against this. Often this is a statement of "no difference."
power
The power of a test against a specific alternative is the probability that the test will reject Ho at a chosen significance level alpha when the specified alternative value of the parameter is true.
power and Type II error
The power of a test against any alternative is 1 minus the probability of a Type II error for that alternative; that is, power = 1 - beta
P-value
The probability, computed assuming Ho is true, that the statistic (such as p-hat or x-bar) would take a value as extreme as or more extreme than the one actually observed. The smaller the P-value, the stronger the evidence against Ho provided by the data.
paired data
The result of study designs that involve making two observations on the same individual, or one observation on each of two similar individuals. This can be analyzed by first taking the difference within each pair to produce a single sample. Then use one-sample t procedures.
significance and Type I error
The significance level alpha of any fixed level test is the probability of a Type I error. That is, alpha is the probability that the test will reject the null hypothesis Ho when Ho is in fact true. Consider the consequences of a Type I error before choosing a significance level.
statistical significance and practical importance
When a null hypothesis ("no effect" or "no difference") can be rejected at the usual levels (α = 0.05 or α = 0.01), there is goof evidence of a difference. But that difference may be very small. When large samples are available, even tiny deviations from the null hypothesis will be significant.
significance test
a formal procedure for comparing observed data with a claim (called a hypothesis) whose truth we want to assess; the claim is a statement about a parameter
Statistical tests ask if sample data give good evidence (for/against) a claim.
against
The smaller the P-value, the great the chance the null hypothesis will (be rejected/fail to be rejected).
be rejected
The hypotheses should express the hopes or suspicions we have (before/after) we see the data. It is cheating to look at the data first and then frame hypotheses to fit what the data show.
before
Failing to find evidence against Ho means only that the data are _______ with Ho, NOT that we have clear evidence that Ho is true.
consistent
When planning a study, verify that the test you plan to use has a (low/high) probability (power) of detecting a difference of the size you hope to find.
high
one-sided alternative hypotheses
if the alternative hypothesis states that a parameter is larger than the null hypothesis or if it states that the parameter is smaller than the null value
two-sided
if the alternative hypothesis states that the parameter is different from the null hypothesis value (it could be either larger or smaller)
Type II error
if you fail to reject the null hypothesis Ho when the null hypothesis Ho is false
Type I error
if you reject the null hypothesis Ho when the null hypothesis Ho is true
Increasing the significance level (increases/decreases) the risk of a Type I error.
increases
Statistical significance (is/is not) the same thing as practical importance.
is NOT
If the df you need isn't provided in Table B, use the next (lower/higher) df that is available.
lower
You should (always/never) "accept Ho" or use language implying that you believe Ho is true.
never
Statistical tests deal with claims about a _______.
population
Always state Ho and Ha in terms of (sample statistics/population parameters).
population parameters
Hypotheses always refer to a (sample/population), NOT to a (sample/population).
population, sample
A test is significant if we (reject/fail to reject) the null hypothesis Ho.
reject
The power of a test to detect a specific alternative is the probability of reaching the (right/wrong) conclusion when that alternative is true.
right
The basic idea of a _______ is this: an outcome that would rarely happen if a claim were true is good evidence that the claim is not true
statistical test
Be sure to report the degrees of freedom with any (z/t) procedure, even if technology doesn't.
t
alternative hypothesis (Ha)
the claim about the population that we are trying to find evidence FOR
The significance level of a test is the probability of reaching the (right/wrong) conclusion when the null hypothesis is true.
wrong