Chapter 21: More About Tests and Intervals

True or False: The P-value of a test does give us the probability that the Null Hypothesis is correct.

False. The P-value is the probability that you will see results like the one you see in the sample that they gave to you to work with, given that the Null Hypothesis is true. In other words, the P-Value is the probability that the observed proportion will happen, given that the Null Hypothesis is true. The P-value is about the data, not the hypothesis.

What is the Effect size?

The Effect size is the difference between the Null Hypothesis value and the true value of a model parameter.

What is the definition of a Significance Level?

The alpha level is also called the significance level, most often in a phrase such as a conclusion that a particular test is "significant at the 5% significance level."

What controls the alpha level in making a Type I Error?

The alpha level is controlled by the level of confidence that you are using to complete a confidence interval for the test.

True or False: The Power of the test depends in part on the sample size. Larger sample sizes lead to greater power and thus fewer Type II Errors.

True. Larger samples allow us to compare results with more consistency and this consistency leads us into the ability to compare our results with a lot more accuracy. Therefore we have a smaller chance of making a Type II Error which is controlled by beta. If beta goes down, then power goes up.

How do we know what type of test to use when setting up a Hypothesis Test?

We need to be able to identify and use the alternative hypothesis when testing hypotheses. When the situation shared insinuates an increase or a decrease in what the Null Hypothesis says, then you would use a one-tail test. But if the situation shared is insinuating a change in what the Null Hypothesis says, then you would use a two-tailed test.

What does statistically significant mean?

When the P-value falls below the alpha level, we say that the test is "statistically significant" at that alpha level.

True or False. We always say that we "accept" the Null Hypothesis when we believe that it is correct.

False. Know that we do not "accept" a Null Hypothesis if we cannot reject it but, rather, that we can only "fail to reject" the hypothesis for lack of evidence against it.

What does it mean to make a Type II Error?

A Type II Error is when you "Fail to reject" the Null Hypothesis when in fact it is actually false. This is also called making a "false negative" conclusion.

What does it mean to make a Type I Error?

A Type I Error is when you "Reject" the Null Hypotheses when you actually should not be rejecting it when in fact it is true. This is also called making a "false positive" conclusion.

What is an Alpha Level?

An Alpha Level is the threshold P-value that determines when we reject a null hypothesis. If we observe a statistic whose P-value based on the null hypothesis is less than alpha, we reject the null hypothesis.

As alpha changes how does that affect the Power of the test?

As alpha goes up, then beta goes down. As beta goes down, then power goes up. And as alpha goes down, then beta goes up. And as beta goes up, then the power of the test goes down. So alpha and power are directly related to each other.

What controls the Beta value in making a Type II Error?

Beta is controlled by alpha. If alpha goes up (which increases the risk of making a Type I Error), then Beta goes down (which is the probability of making a Type II Error). And if alpha goes down, then Beta goes up. These two probabilities are inversely related.

What is Power of the test?

Power is the probability of "Rejecting" the Null Hypothesis when you actually should be rejecting it. The Null Hypothesis is actually false and you are correctly rejecting it because it is false.

What is the probability of making a Type I Error called?

The probability of making a Type I Error is called alpha.

What is the probability of making a Type II Error called?

The probability of making a Type II Error is called Beta and depends on the effect size.

How doe we calculate the Power of the test?

To find the Power, we must specify a particular alternative parameter value as the "true" value. For any specific value in the alternative, the Power is found by taking 1 - beta.