Statistics- Significant levels, p-Values

statisitical significance

- when your p-value is less than α( the significance level), your results are statistically significant -this condition indicates that the strength of the evidence in your sample (p-value) exceeds the evidentiary standard you defined prior to the study (significance level) - the sample evidence provides sufficient evidence to conclude that the effect exists in the population

p-value

Serves as a metric for evaluating the strength of evidence against a null hypothesis - derived from the sample statistic and is subsequently compared to the alpha level to decide whether to accept or reject null hypothesis - quantifies the evidence against the null hypothesis. It represents the probability of obtaining the observed or more extreme results, assuming the null hypothesis is true

Two new COVID meds (Errors)

- Null hypothesis (H0): the two meds are equally effective - Alternative hypothesis (H1): the two meds are not equally effective - Type I error: Rejects H0, when it is true (the patients still benefit from the same level of effectiveness) - Type II error: Fails to reject H0, when it should be rejected (if the less effective med is sold to the public instead of the more effective one)

Alpha (α)

- Pre-determined level of significance chosen to make decisions in a hypothesis test - It defines the probability of making a Type 1 error, which is the error of rejecting a true null hypothesis - Sets the threshold for how strong the evidence against the null hypothesis must be before rejecting it

Test Statistic (TS or xi)

- calculated point statistic - random variable (e.g. mean) that is calculated from the sample data used to determine how different the finding is from the alpha level - used to determine whether to reject or accept the null hypothesis - used to calculate the p-value - observed value of a test statistic changes randomly from one random sample to another

Type II Error (False Negative) (beta)

- denoted by beta (beta error) - occurs when the null hypothesis is accepted when it is not true - results in a false negative result -failing to accept an alternative hypothesis when the researcher doesnt have adequate power

Type 1 error (false positive)

- denoted by α, the designated region of the PDF - known as alpha error - this type of error is a false positive error - called the level of significance of the tests -occurs when the null hypothesis that should have been accepted is rejected or when there is no relationship between the variables

how to improve the power of a study

- increase sample size - increase difference between groups (increase effect size; the actual difference between the means) - increase precision of results (decrease SD)

Alpha defines

- the critical value on a PDF - Critical Values z, t, F, x2 defines the rejection region - it is chosen before starting the research - alpha is the smallest value of p that we are willing to accept and say that there is a difference, a change and there is an effect

p-value stats

- the p-value is a probability value derived from the sample statistic - the p-value is a number between 0 and 1 - the p-value is compared to alpha region on the PDF

Type I and Type II Errors

- the risk of these two errors are inversely related - determined by the level of significance and the power of the tests

α-level

- the significance level or threshold the research sets before conducting a hypothesis test - represents the maximum allowable probability for making a Type I error - Commonly used alpha levels are 0.05 (5%) and 0.01 (1%)

Increase power?

- type II error decreases inversely in a statistical test - as the power increases, the probability of a type II error decreases

p value less than or equal to alpha (<0.05)

indicates strong evidence against the null hypothesis, so you reject the hypothesis. it is deemed statistically significant

smaller p value

indicates stronger evidence against the null hypothesis, signify greater significance in the result

p value greater than alpha (>0.05)

indicates weak evidence against the null hypothesis, so you accept the null hypothesis

Statistically Significant Result

means that the threshold (you set the threshold) for rejecting the null hypothesis has been met

Statistical significance results

p value

TS > Critical Value

p-value < α - this is the criteria for the level of confidence where you reject the null hypothesis(p-value)

TS < Critical value

p-value > α - means not significant, accept the null hypothesis

Statistically significant results

will not prove that a research hypothesis is 100% correct - there is always a 5% probability it occurred by chance

Probability/Alpha level

α = P(rejecting the null hypothesis / the null hypothesis is true)

alpha& p-value

Alpha and p-values are both values based on the assumed or known PDF (probability density function)

Inferential statistics?

Evidence against the Null hypothesis is provided by demonstrating that the Alternative Hypothesis is true - We always want to compare the alternative hypothesis to the null hypothesis - We want the null hypothesis to be TRUE

Test statistic is PDF

If the TS is in the α region, the random variable calculated from the sample data is statistically significant

Can we prove something is true?

Never. Instead, we find Evidence something isn't true. You need to observe the outcome of every single possible scenario to provide evidence to the contrary

P< 0.05 is the accepted cut-off by Convention

Ronald Fisher - he arbituarily decided that 0.05 should be the cut-off value for accepting or rejecting the null hypothesis

p-value and evidence

The p-value is an arbitrary cut-off value used to compare the test statistic to the Alpha (α) to determine if it is a statistically significant result

p-value and data analysis

There is always a corresponding p-value for any statistical test used for data analysis - The smaller the p-value, the greater the confidence for rejecting the null hypothesis

Four possible outcomes in inferential statistics

Two correct decisions and two errors

Critical value - one tail hypothesis

X~ N(0,1) total probability 1-alpha

Critical value- two tail hypothesis

X~ N(0,1) total probability 1-alpha critical value= α/2

Decrease risk of type-2 error

ensure that your test has enough power by confirming that the sample size is large enough to detect a practical difference when one truly exists

Result of Type II Error

researcher may end up believing that the alternate hypothesis is not significant, when it is

Type I error

the incorrect rejection of a true null hypothesis (a "false positive")

Smaller the alpha level

the more strict the test is, and the less likely it is to reject the null hypothesis when it is true incorrectly - also increases the chance of failing to reject the null hypothesis when it is false ( Type II error or false negative)

If the p-value is low...

the null must go - the data is actually statistically signficant

power (1- beta)

the probability of detecting a true effect if one exists

Result of Type 1 error

the researcher might believe that the hypothesis works even when it doesn't

Critical value (diamond)

the value on the PDF defining the probability of the rejection region(α)