Statistics Unit 3: Hypothesis Testing, One Sample t-Tests, Paired- Samples t-Tests, Independent-Samples t-Tests
What does Type 1 error mean?
Incorrect Rejection; ReJecting a null hypothesis when it is actually true; concluding that there is a significant effect - based on our data - when, in reality, there is NO true effect.
What does Type II error mean?
Incorrect Retention; ReTaining a null hypothesis that should be rejected; concluding that there is no significant effect - based on our data - when, in reality, there is a true effect.
When is a t test appropriate to use, instead of a z test?
When we do not have information about the population standard deviation and/or have a small sample size
How is the alpha level important in NHST?
It helps us decide whether or not we should reject the null hypothesis
Why is effect size useful?
You can look at the effect size when comparing any two groups to see how substantially different they are.
What is NHST?
a method of statistical inference by which an experimental factor is tested against a hypothesis of no effect or no relationship based on a given observation, null hypothesis significance testing
How can a confidence interval be used to test a null hypothesis?
o If zero does not fall between the LL and the UL, then we can be 95% confident that the difference between u1 and u2 is not zero. o Therefore, confidence interval can also be used the test the null hypothesis: If zero falls between the LL and the UL, the retain. If zero does not fall between the LL and the UL, the reject.
Decision rule: If the test statistic falls past the critical value...
reject H0
Decision rule: If the test statistic does not fall past the critical value...
retain H0
What is the rejection region?
the area of a theoretical distribution that corresponds to the rejection of the null hypothesis and includes the differences that have a probability equal to or less than alpha.
What is the standard error of the difference?
the probability that a difference between the statistical means of two samples is greater than zero; standard error of the difference between two means
What are critical values?
the values of the test statistic that determine the "cutoff" points for the rejection region.
How do the steps of NHST for t tests differ from NHST for z tests?
you don't have sample sizes for z tests, so you would not use degrees of freedom in the steps of NHST for z tests.
When will we be likely to use a one-sample t test?
• Checking a sample versus an advertised claim • Comparing a sample to a known norm (e.g., population mean) • Comparing behavior against a "no error" standard (e.g., studying lying behavior or visual illusions)
What does it mean to retain the null hypothesis?
• This does NOT mean we have proven the null hypothesis is true. • While the null hypothesis might be true, it is also possible that our research design was not sensitive enough to detect the true effect.
How do you interpret the output from jamovi?
"A one-sample t test revealed that emerging adults have significantly higher levels of perceived stress scores (M=30.30 SD=6.79) than the US adult population average (u=14.2), t(196)=32.50, p < 0.001, d = 2.31, 95% CI=[15.1, 17.1]. The magnitude of this effect was large"
What does a strong conclusion for a paired-samples t test look like?
"A paired samples t test revealed the severity of asthma symptoms following relaxation training (M=1.80) is significantly lower than severity of asthma symptoms prior to relaxation training (M=5.80), t(4)=4.49, p=0.006, d=2.00, 95% CI=[2.09, à]. The magnitude of this effect is very large."
What does a strong conclusion for an independent samples t test look like?
"According to an independent-samples t test... we conclude that using mental images to learn word pairs produced memory performance (M=25.00) that is significantly higher than learning word pairs without mental images (M=19.00), t(18)=-3.00, p=0.008, d=-1.34, 95% CI=[-10.20, -1.80]. The magnitude of this effect is very large."
How do we interpret alpha?
"Assuming the null hypothesis is true, we are willing to conclude that the difference between our sample and population means is statistically significant if there is less than a ___ probability of its occurrence."
How can we interpret a confidence interval for a paired-samples t test?
"Based on our sample data, we can be 95% confident that the true difference between the population means falls between ___ (UL) and ___ (LL)."
How do we interpret confidence interval for an independent samples t test?
"Based on our sample data, we can be ___% confident (1-α) that the true difference between the population means falls between LL and UL."
How do you write up a null hypothesis with words?
"do not significantly differ"
How do you write up an alternative hypothesis with words?
"do significantly differ"
What is the alpha level?
(significance level); the probability of a Type I Error. typically set at .05 or .01.
How does the Standard Error of the Difference affect power?
- As its value gets smaller, t gets bigger, thus making it more likely that we will reject the null hypothesis. - Bigger sample size = smaller standard error of the difference. - Smaller sample size = smaller standard error of the difference. - All about sampling variability.
What 3 factors affect power?
- Effect size - Alpha - Standard Error of the Difference
What ways are the "New Statistics" considered an improvement upon the "Old Statistics?"
- Increased reliance on Confidence Intervals, because they provide both point and interval estimates - Increased reliance on Effect Sizes (a way of quantifying how different two things are)
What is the difference between one-sample and two-sample t tests?
- One-sample t tests compare a score or a sample mean to the population. - Two-sample t tests compare two different samples to one another.
What is the difference between paired- and independent-samples t tests?
- Paired-samples designs have two groups of participants that are 'matched' in some way OR have repeated measures. - Independent-samples designs have separate, unmatched samples for each group of participants (between-subjects design).
What are the advantages of using a paired-samples design over an independent samples design?
- Reduces error due to the individual differences between samples-- smaller denominator for t test - Increased power (our ability to detect a real difference); reducing the difference/variability
What is a paired-sample t test with repeated measures?
- experimental design where each participant completes more than one measure. - both scores belong to the same participant - uses fewer participants, more economical and allow for studies with rare participants - examines change over time
What is a paired-sample t test with matched pairs?
- individuals are paired by the researcher before the experiment. - based on skill or specific measurement ahead of time
What is a paired-sample t test with natural pairs?
- pairing that occurs naturally and without intervention by the experimenter. - ex. "Participant 20 in my experimental group is matched with Participant 20 in my control group."
What is power?
- the probability of rejecting the null hypothesis when, in fact, it is false. -the probability of making a correct decision (to reject the null hypothesis) when the null hypothesis is false. - the probability that a test of significance will pick up on an effect that is present. - the probability that a test of significance will detect a deviation from the null hypothesis, should such a deviation exist. - the probability of avoiding a Type II error. Simply put, power is the probability of not making a Type II error
How can we compute a confidence interval for an independent-samples t test?
- using sample data to estimate the value of the true population mean difference
What are the elements of a good conclusion paragraph?
-Findings placed in context. -Descriptive and inferential statistics provided and interpreted.
Why wouldn't it be a good idea to reduce alpha to something far less than .05 or .01?
-If we reduced the value of alpha to something far less than .05 or .01, we would increase the value of beta, which would then increase the risk of committing a type II error.
What criticisms exist about NHST?
1. Based on faulty assumption; the null hypothesis is never true 2. Results in dichotomous (yes/no) conclusions 3. p values are often misused/misunderstood 4. p. values can change very easily
What does effect size tell us?
Effect size is a quantitative measure of the magnitude of the experimenter effect. The larger the effect size the stronger the relationship between two variables. You can look at the effect size when comparing any two groups to see how substantially different they are.
What is an independent samples t test?
Experimental design with samples whose dependent variable scores cannot logically be paired
What do alternative hypotheses look like?
Hypothesis of difference; A difference that is not likely due to chance, the difference is "real"
What do null hypotheses look like?
Hypothesis of equality; no difference between two things is being compared; comparing sample mean to population mean
What is a paired sample t test?
Two sample t test; Experimental design in which scores from each group are statistically/logically matched
What is the difference between one-tailed and two-tailed tests?
Two-tailed (non-directional) tests can detect positive or negative differences. One-tailed (directional) tests can detect either a positive or a negative difference, but not both.
What are the potential errors in hypothesis testing?
Type I and Type II
How can we compute a confidence interval for a paired-samples t test?
Use confidence intervals to estimate the true difference between the populations that our samples represent.
What does it mean to say that, in NHST, we are trying to determine if a particular €score or sample mean is from the population of interest or a different population?
We are determining whether or not to reject the null hypothesis, and if that particular score is in the rejection region, we may infer that it may be part of a different population from the population of interest.
What is a confidence interval?
A confidence interval (CI) is a range of values that's likely to include a population value with a certain degree of confidence. It is often expressed a % whereby a population means lies between an upper and lower interval.
How do we determine where the rejection region(s) will be for any given NHST?
By the critical values.
How does alpha relate to the critical value?
Critical value is the t-value showing where alpha sits on the distribution marking off the rejection region. In other words, alpha provides the probability of a particular score falling behind our critical value/cutoff point.
What is a one-tailed test?
Directional tests; can detect positive or negative differences, but not both; places alpha on one tail of the distribution
What are the recommended "New Statistics"?
NHST with a lot of emphasis on Effect Sizes and Confidence Intervals and writing strong, up-to-date conclusions
What are the three types of paired-sample designs?
Natural pairs, Matched pairs, and Repeated Measures
What is a two-tailed test?
Non-directional; can detect positive or negative differences; divide alpha between the two tails of the distribution
What are the two categories of hypotheses in NHST?
Null Hypothesis & Alternative Hypothesis
What is a one sample t-test?
Statistical test of the hypothesis that a sample with mean "X bar" came from a population with a mean μ; Comparing a sample mean to the population
What are the 5 steps involved in NHST?
Step 1: Specify appropriate test and justify Step 2: Formulating hypotheses Step 3: Specify sample size, df, and alpha, locate the critical value for your test Step 4: Calculate your test statistics Step 5: Make a decision about H0 and write up your results
How does Alpha affect power?
The larger the alpha is, the more likely to reject the null hypothesis.
How does Effect Size affect power?
The larger the effect size, the more likely it is that you will reject the null hypothesis.
How are the two errors in NHST related to one another?
They are inversely related to each other
What does it mean to reject the null hypothesis?
This could mean: • There is a real difference between our sample and the population, or a particular score and the population .• There is not a real difference present; we got the results as a result of natural sample variability. • We have a biased sample.
How do we interpret effect size?
d = 0.20 -> small effect d = 0.50 -> medium effect d = 0.80 -> large effect