Chapter 6 - Testing for Statistical Significance

Binomial test - interpreting Stata output and meaning of test

- Compare experiment's success rate to a standard rate - Only two possible outcomes - Calculated by Stata rather than running simulations - All observations must be independent

Computer simulation

- Enter data into Stata with half of population gain mass (1) and half do not gain mass (0) - Have Stata select 100 random samples of 20 bodybuilders - Results will show if it is unusual to find 65% gain mass when 50% is true rate

Sample

- Group selected within population - Different samples will give different results

The Effect of Sample Size

- What if you use samples of 60 bodybuilders instead of 20 with population muscle mass change of 50% without supplements? - As the sample size increases, the variability of statistics calculated from the sample decreases

Statistic

A characteristic of a sample

Parameter

A characteristic of the population

Alpha level or P(critical) - Definitions

Alpha Level: Before interpreting statistical tests, scientists or researchers set an alpha level, which is also referred to as P(critical). The alpha level is the probability of rejecting the null hypothesis when it is true. It is typically set at 0.05 of 5%, but researchers also use 0.01, 0.001 and sometimes 0.1. The larger the alpha level, the more likely you are to find statistically significant results. P-Value: Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data (e.g., the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.

Statistical significance decision rule

Decision Rule: • If p-value <= alpha level, reject the null • If p-value > alpha level, do not reject the null hypothesis Our Results: • p-value = .24 • alpha level = .05 Decision: • p-value (.24) > alpha level (.05) • Do not reject the null hypothesis In other words, in 24 percent of all samples, we saw muscle mass changes at the extreme values. Since this is fairly high (much higher than five percent), we would say that it is not that unusual to see high or low gains in muscle mass without taking the supplements. Therefore, we would not believe the manufacturer's claim.

Bar graph of a sampling distribution (interpretation of numbers, meaning, generating p-value from bar graph)

Figure 6.1 on pg 27 of textbook Sampling distribution (probability distribution): the set of numbers at the top of each bar is the "Sampling distribution (probability distribution)" of a statistic. - 100 samples - the numbers at the top of the bars, or the frequencies, add up to 100 since we drew from 100 samples. - Each sample has 20 bodybuilders (1-gained mass, 2-did not gain mass) - Outcome: total number that gained mass out of 20 bodybuilders - Percent Success: % of 20 bodybuilders that gained mass in each sample - We can expect to see an increase in mass in at least 65% of bodybuilders in 14% of all samples - We can expect to see an increase in mass in 35% or less of bodybuilders in 10% of all samples - 24% of samples fall in the two extremes (>=65% and <=35%)

Random sample

Gives every member of the population the same chance of being included in sample

Population

Group you are interested in

Standard Deviation

How much statistic varies in sample

Statistical significance decision rule - Implications

It is important to note that we would never say, "we accept" the null hypothesis since there is always some chance that our samples did not accurately reflect the population. In fact, the alpha level tells us the probability of rejecting the null hypothesis when it is true. This is referred to as a type I error. A type II error occurs when we do not reject the null when it is false. Recently, the American Statistical Association released a statement on statistical significance and p-values to correct some of the many misuses of the concept. As they emphasize, the p-value does not indicate if a hypothesis is true or if the data were produced by random chance. Furthermore, they emphasize that researchers should consider other factors besides the p-values such as the "design of a study, the quality of the measurements, the external evidence for the phenomenon under study, and the validity of assumptions that underlie the data analysis." Finally, they suggest other methods in addition to p-values to test hypotheses, such as confidence intervals, which are discussed in later chapters.

Standard Error

Standard deviation of the distribution of all possible values of a statistic E.g., standard error of the mean is the standard deviation of all possible values of a sample mean

Sampling distribution

The distribution of all possible sample outcomes for a statistic

Comparing p-value to the alpha level

Using our alpha level of five percent, we would say that the manufacturer's claim is statistically significant if five percent or fewer of repeated examples exhibited extreme values of muscle mass changes (greater than or equal to 65 percent or less than or equal to 35 percent). In our example above, 24 percent of the samples fell in the two extremes. This is called our p-value. We can now compare our p-value to the alpha level to determine whether our results are unusual or statistically significant. The rule along with our example is shown in Figure 6.2.

Formulating the null hypothesis from a research question

When using statistical tests, however, we would define a null hypothesis, which is a testable statement indicating that there is no difference or no change. Research Question: Do a larger portion of bodybuilders gain mass with the new supplement compared to those who do not take it? Research hypothesis or null hypothesis: A larger proportion of bodybuilders do not gain muscle mass when taking the new supplement.