p-values

T or F: a large p-value eventually becomes of the null hypothesis

False. Any p-value less than 1.0 implies that the null hypothesis is not the best hypothesis to explain the given data. A p-value cannot said to favour the null hypothesis except in relation to those hypothesis with smaller p-value.

T or F: If you reject the test hypothesis because P =< 0.05, the chance you are in error (the chance your ''significant finding'' is a false positive) is 5 %.

False. The 5% only refers to how often you reject it, it does not refer to your single use of the test.

T or F: a significant test result means that the null hypothesis is false and should be rejected

False. A p =< .05 only flags the data as being unusual given the assumptions used to computed were true. (remember that it may also be the case that the p value was small due to random error or violation of assumptions)

T or F: a non-significant test result means the null hypothesis should be accepted.

False. A large (p value > .o5) only suggests that the data are what we would expect if all the assumptions were true.

T or F: a large null p-value flags the data as not being unusual and the same data will be unusual under many other models.

False. A large null P value simply flags the data as not being unusual if all the assumptions used to compute it (including the test hypothesis) were correct; but the same data will also not be unusual under many other models and hypotheses besides the null.

T or F: P value may be very small or very large depending on whether the study and the violations are large or small.

True.

T or F: The lack of statistical significance of individual studies should not be taken as implying that the totality of evidence supports no effect.

True.

What does the p-values distribution look like when the null is true?

distribution is flat. All p-values are equally likely to be true

T or F: A null-hypothesis P value greater than 0.05 means that no effect was observed, or that absence of an effect was shown or demonstrated.

False. It only means that the null is one of the many hypothesis > 0.05. If the p value is less than 1.0, some association must be present in the data, and we look at the point estimate to determine effect size most compatible with the data under the assumed model.

T or F: When the same hypothesis is tested in two different populations and the resulting P values are on opposite sides of 0.05, the results are conflicting.

False. Statistical tests are sensitive to many differences between study populations. As a consequence, two studies may provide very different P values for the same test hypothesis and yet be in perfect agreement.

T or F: The P value is the probability that the null hypothesis is true

False. The p value always assumes the null hypothesis is true (remember that it is not a hypothesis probability). The p-value simply indicates the degree to which the data conform to the pattern predicted by the test hypothesis

T or F: When the same hypothesis is tested in two different populations and the same P values are obtained, the results are in agreement.

False. Two different studies could exhibit identical p-values for testing the same hypothesis, yet also exhibit clearly different observed associations.

What does the p-value distribution look like when the null is false?

The distribution is positively skewed. More likely to get p-values closer to zero.

T or F: Two-sided tests will falsely reject the null only half as often as the one-sided test will falsely reject the alternative.

True.

T or F: There is little practical difference among very small P values when the assumptions used to compute P values are not known with enough certainty

True. Most methods for computing p values are not numerically accurate below a certain point.

T or F: The p-value is deduced from a set of assumptions and does not refer to the probability of those assumptions.

True. The p-value is a probability computed that assumes chance was operating.

T or F: p = 0.05 is considered a borderline result.

True. This is why we say p=< in significance testing, because then it lumps borderline results together with results very incompatible with the null hypothesis.

T or F: lack of statistical significance does not indicate a small effect size.

True. We look at the confidence intervals to determine whether it includes effect sizes of importance. Even large effect sizes can be "drowned" in noise and fail to be detected.