Stats Chapter 3

NHST assumes that we have a random sample of N independent observations from the population of interest and that the scores on the X variable are quantitative and at least approximately interval/ratio level of measurement. These are not always the case. Mistaken assumption that if Ho is rejected, one affirms the theory that led to the test.

Describe at least two potential problems with NHST.

To check for "practical" or "clinical" significance, we check if the sample test statistic falls in the rejection region or not, or whether or not the p-value of the test is less than the level of significance of the test. If the test statistic belongs to the rejection region or the p-value is less than the level of significance, it is concluded that the result is "practically" or clinically" significant.

Briefly discuss: What information do you look at to evaluate whether an effect obtained in an experiment is large enough to have "practical" or "clinical" significance?

Statistical Power is the probability (1-β) of rejecting null hypothesis when it is false, and this null hypothesis should be rejected in order to avoid Type II error. Therefore, one needs to keep the Statistical Power correspondingly high, as the higher our Statistical Power, the fewer Type II errors we can expect.

How are the risk of a Type II error and the statistical power related?

Statistical significance, often represented by the term p < .05, has a very straightforward meaning. If a finding is said to be "statistically significant," that simply means that the pattern of findings found in a study is likely to generalize to the broader population of interest.

In your own words, what does it mean to say "p < .05"?

A non-directional test or two-tailed test requires a value of t that is larger in absolute value to reject HO compared to a directional test. A directional test or one-tailed test is set at a value above OR below at +/- 1.645. A non-directional test or two-tailed test is set at a value above and below at +/- 1.965.

Other factors being equal, which type of significance test requires a value of t that is larger (in absolute value) to reject H0—a directional or a nondirectional test?

The statement is not correct because, the research is only based on one sample it is evident that only 3.2% of the times, the null hypothesis is correct, whereas, based on some other sample, there might be another value coming up.

Suppose a researcher writes in a journal article that "the obtained p was p = .032; thus, there is only a 3.2% chance that the null hypothesis is correct." Is this a correct or incorrect statement?

The nominal alpha level sets the limit on the magnitude of risk of Type I error. The risk of committing the error is the alpha level. Example: alpha level = .05, the risk of committing error is 5%.

What research decision influences the magnitude of risk of a Type I error?

We typically want "p" to be small so that the probability of committing Type I error is low.

When a researcher reports a p value, "p" stands for "probability" or risk. Do we typically want p to be large or small?

If a student is viewed as "statistically significant", this means that the sample data provides enough evidence against the null hypothesis, therefore, the null is to be rejected in favor of the alternative hypothesis.

What conclusions can be drawn from a study with a "statistically significant" result?

If the results of a study is "null", it means that the experimenter fails to reject the null hypothesis of the study. This cannot always mean that the null hypothesis is definitely true because Type I errors can occur. Therefore, when null result is obtained, the conclusion is that there isn't sufficient evidence against the null and therefore it is not rejected.

What conclusions can be drawn from a study with a null result?

The factors that influence type II error are: 1 - Sample size of the research. As sample size increases, Type II error should reduce. 2- Pre-set alpha level by the researcher. Smaller set alpha level the larger risk of a Type II error. 3- True Population effect size (usually measured as Cohen's d). As true population effect increase, Type II error should reduce.

What factors influence the magnitude of risk of a Type II error?

The null hypothesis is a statement about a population parameter. We test the likelihood of this statement being true in order to reject or not reject the null hypothesis. The alternative hypothesis directly contradicts the null hypothesis. We decide to reject or not reject the alternative hypothesis based on our null hypothesis test.

What is a null hypothesis? An alternative hypothesis?

The exact p value is the (theoretical) probability of obtaining a sample mean, M, farther away from the hypothesized value of μ specified in the null hypothesis than the value of M in the sample in the study, if H0 is actually correct.

What is an "exact" p value?

Before you run any statistical test, you must first determine your alpha level, which is also called the "significance level." By definition, the alpha level is the probability of rejecting the null hypothesis when the null hypothesis is true. Alpha is usually 0.05. The value of alpha is pre-set by the researcher based on the amount of risk he is willing to take during the hypothesis testing procedure.

What is an alpha level? What determines the value of α?

Statistical power is defined as the probability of rejecting the incorrect null hypothesis. To determine the specific sample size required for statistical power, we need to know alpha level and the population effect size.

What is statistical power? What information is needed to decide what sample size is required to obtain some desired level of power (such as 80%)?

A directional test or single-tailed test has the reject region on one end of the distribution. > means that it will be on the right side and < it will be on the left side. A directional test or one-tailed test is set at a value above OR below at +/- 1.645. A non-directional test or two-tailed test requires a value of t that is larger in absolute value to reject HO compared to a directional test. A non-directional test or two-tailed test is set at a value above and below at +/- 1.965.

What is the difference between a directional and a nondirectional significance test?

The APA now recommends that we do not report significant test results alone. In addition to statistical significance tests we should report descriptive data for all groups, confidence Intervals; and effect size information.

What recommendations did the APA Task Force (Wilkinson & Task Force on Statistical Inference, 1999) make about reporting statistical results? Are significance tests alone sufficient?

The conventional standard is that the "p-value" should be lesser than 0.05, then the value is considered "acceptably small" for null hypothesis to be rejected.

When a researcher reports a p value, "p" stands for "probability" or risk. What is the conventional standard for an "acceptably small" p value?

The value "p" is the actual/statistical probability of committing Type I error in the test.

When a researcher reports a p value, "p" stands for "probability" or risk. What probability does this p refer to?

In real life situations usually, the assumptions of NHST are violated, and when they are violated, the nominal p values usually underestimate the Type I error.

Why do reported or "nominal" p values often seriously underestimate the true risk of a Type I error?