QM Exam 3

TEST QUESTION, name one of them

dependent t-tests are also referred to as these types of test: repeated measures t test within subjects t test paired t test include the word t test****

Role of Power When a Result is Not Statistically Significant

• A nonsignificant result from a study with low power is truly inconclusive. • A nonsignificant result from a study with high power suggests that: - the research hypothesis is false or - there is less of an effect than was predicted when calculating power

Planning a Sample Size

• A power table can be used to see how many participants you would need to have enough power. - Many studies use 80% as the power needed to make the study worth conducting. - To use a power table to determine the number of participants needed in a sample: • Decide whether you need a one- or a two-tailed test. • Determine the expected effect size. • Determine the level of power you want to achieve (usually .80). • Use this information to guide you to the appropriate columns and rows on the power table.

The Standard Deviation of the Distribution of Means

• After finding the estimated population variance, you can calculate the standard deviation of the comparison distribution. - The variance of a distribution of means is the variance of the population of individuals divided by the sample size. - The standard deviation of the distribution of means based on an estimated population variance is the square root of the variance of the distribution of means based on an estimated population variance. - S is used instead of Population SD when the population variance is estimated. S^2M = S^2 / N SM = √S^2M

Effect Size

• An effect can be statistically significant without having much practical significance. • Effect Size - It is a measure of the difference between populations. - It tells us how much something changes after a specific intervention. - It indicates the extent to which two populations do not overlap. • how much populations are separated due to the experimental procedure - With a smaller effect size, the populations will overlap more.

Effect Size and Power in Research Articles

• Articles often mention effect size. • Effect size is a crucial factor in metaanalyses, and thus is almost always reported in meta-analyses. • Power is sometimes discussed when evaluating nonsignificant results.

Assumptions of the t Test for a Single Sample and t Test for Dependent Means

• Assumption - a condition required for carrying out a particular hypothesis-testing procedure - It is part of the mathematical foundation for the accuracy of the tables used in determining cutoff values. • A normal population distribution is an assumption of the t test. - It is a requirement within the logic and mathematics for a t test. - It is a requirement that must be met for the t test to be accurate.

Relationship Between Type I and Type II Errors

• Decreasing the probability of a Type I error increases the probability of a Type II error. - The compromise is to use standard significance levels of p < .05 and p < .01.

When Using a t Table...

• Determine whether you have a one- or a two-tailed test. • If you are using a one-tailed test, decide whether your cutoff score is a positive or a negative t score. - If your one-tailed test is testing whether the mean of Population 1 is greater than the mean of Population 2, the cutoff t score is positive. - If the one-tailed test is testing whether the mean of Population 1 is less than the mean of Population 2, the cutoff t score is negative. • Decide which significance level you will use. • Find the column labeled with the significance level you are using. • Go down to the row for the appropriate degrees of freedom. • If your study has degrees of freedom between two of the higher values on the table, you should use the degree of freedom that is nearest to yours and less than yours.

Formula for Calculating the Effect Size

• Effect Size = Population 1 M - Population 2 M/ Population SD - Population 1 M = the mean for the population that receives the experimental manipulation - Population 2 M = the mean of the known population (the basis for the comparison distribution) - Population SD = the standard deviation of the population of individuals - A negative effect size would mean that the mean of Population 1 is lower than the mean of Population 2.

Key Points

• Effect size is a measure of the difference between population means. It is figured by dividing the difference between population means by the population standard deviation. • Small effect size = .20, Medium = .50, Large =.80 • Statistical power is the probability that the study will produce a statistically significant result if the research hypothesis is true. • The larger the effect size, the greater the power. • The larger the sample size, the greater the power. • Power is also affected by significance level, whether a one- or twotailed test is used, and the type of hypothesis-testing procedure. • Statistically significant results from a study with high power may not have practical importance. • Non-significant results from a study with low power are inconclusive. • Research articles often report effect size, and effect sizes are always discussed in meta-analyses. • Power is sometimes discussed in research articles.

The Role of Power When Planning a Study

• Even if the research hypothesis is true, if you conduct a study that has low power, your results will most likely not be statistically significant. • There are some practical ways a researcher can increase the power of a study. - Increase the effect size by increasing the predicted difference between population means. • Use a more intense experimental procedure. • Elaborate on the instructions in the experimental condition. • Methods to increase the impact of an experimental procedure can be difficult or costly and may lead you to use a procedure that is very different from the situation to which you would like to generalize your results. - Increase the effect size by decreasing the population standard deviation. • Use a population that has less variation within it than the one you planned to sample. - The disadvantage to this is that the results will then only apply to the more limited population. • Use more standardized testing conditions and more precise measures. - Test in a controlled laboratory setting. - Use measures and tests with very clear wording. - Increase the sample size. • This is the main way to change a planned study to raise its power. • There is a limit to how many participants are available. • A larger sample adds to the time and cost of conducting research. - Use a less extreme level of significance. • Using a level of significance that is less extreme than p < .05 is not recommended because this increases the chance of making a Type I error. - Use a one-tailed test. • It is rarely possible to change this, as it is dependent on the logic of the hypothesis being studied. - Use a more sensitive hypothesis-testing procedure. • It is rarely possible to change this, as researchers generally attempt to use the most sensitive hypothesistesting procedure available to them.

Difference Scores

• For each person, you subtract one score from the other. • If the difference compares before versus after, difference scores are also called change scores. • Once you have the difference score for each person in the study, you do the rest of the hypothesis testing with difference scores. - You treat the study as if there were a single sample of scores.

Example of Calculating the Effect Size

• For the sample of 64 fifth graders, the best estimate of the Population 1 mean is the sample mean of 220. • The mean of Population 2 = 200 and the standard deviation is 48. • Effect Size = Population 1 M - Population 2 M/ Population SD • Effect Size = 220 - 200/ 48 • Effect Size = .42

t Tests

• Hypothesis-testing procedure in which the population variance is unknown - compares t scores from a sample to a comparison distribution called a t distribution • t Test for a single sample - hypothesis-testing procedure in which a sample mean is being compared to a known population mean but the population variance is unknown - Works basically the same way as a Z test, but: • because the population variance is unknown, with a t test you have to estimate the population variance • With an estimated variance, the shape of the distribution is not normal, so a special table is used to find cutoff scores.

t Tests for Independent Means

• Hypothesis-testing procedure used for studies with two sets of scores - Each set of scores is from an entirely different group of people and the population variance is not known. • e.g., a study that compares a treatment group to a control group

Effect Size Example

• If Population 1 had a mean of 90, Population 2 had a mean of 50, and the population standard deviation was 20, the effect size would be: - (90 - 50) / 20 = 2 • This indicates that the effect of the experimental manipulation (e.g., reading program) is to increase the scores (e.g., reading level) by 2 standard deviations.

Role of Significance in Sample Size When Interpreting Research Results

• If the result is statistically significant and the sample size is small, the result is important. • If the result is statistically significant and the sample size is large, the result might or might not have practical implications. • If the result is not statistically significant and the sample size is small, the result is inconclusive. • If the result is not statistically significant and the sample size is large, the research hypothesis is probably false.

Effect Size and Power

• If there is a is a mean difference in the population, you have more chance of getting a significant result in the study. - If you predict a bigger mean difference, the power based on that prediction will be greater. - Since the difference between population means is the main component of effect size, the bigger the effect size, the greater the power. - Effect size is also determined by the standard deviation of a population. • The smaller the standard deviation, the bigger the effect size. - The smaller the standard deviation, the greater the power.

The t Test for Dependent Means

• It is common when conducting research to have two sets of scores and not to know the mean of the population. • Repeated Measures Design - research design in which each person is tested more than once - For this type of design, a t test for dependent means is used. • The means for each group of scores are from the same people and are dependent on each other. • A t test for dependent means is calculated the same way as a t test for a single sample; however: - Difference scores are used . - You assume that the population mean is 0.

A More General Importance of Effect Size

• Knowing the effect size of a study lets you compare results with effect sizes found in other studies, even when the other studies have different population standard deviations. - Knowing the effect size lets you compare studies using different measures, even if the measures have different means and variances. • Knowing what is a small or a large effect size helps you evaluate the overall importance of a result. - A result may be statistically significant without having a very large effect. • Meta-Analysis - a procedure that combines results from different studies, even results using different methods or measurements - This is a quantitative rather than a qualitative review of the literature. - Effect sizes are a crucial part of this procedure.

Effect Size for the t Test for Dependent Means

• Mean of the difference scores divided by the estimated standard deviation of the population of difference scores estimated effect size = M/S M = mean of the difference scores S = estimated standard deviation of the population of individual difference scores

Population of Difference Scores with a Mean of 0

• Null hypothesis in a repeated measured design - On average, there is no difference between the two groups of scores. • When working with difference scores, you compare the population of difference scores from which your sample of difference scores comes (Population 1) to a population of difference scores (Population 2) with a mean of 0.

Power

• Power for a t test of dependent means can be calculated using a power software program, a power calculator, or a power table. • Table 8-9 in your textbook shows an example of a power table for a .05 significance level. • To use a power table: - Decide whether you need a one- or a two-tailed test. - Determine from previous research what effect size (small, medium, or large) you might expect from your study. - Determine what sample size you plan to have. - Look up what level of power you can expect given the planned sample size, the expected effect size, and whether you will use a one- or a two-tailed test.

Statistical Power

• Probability that the study will produce a statistically significant result if the research hypothesis is true - When a study has only a small chance of being significant even if the research hypothesis is true, the study has low power. - When a study has a high chance of being significant when the study hypothesis is actually true, the study has high power. • Statistical power helps you determine how many participants you need. • Understanding power helps you make sense of the results that are not significant or results that are statistically significant but not of practical importance. • To determine power, researchers can use a power software package or a power calculator. • Power can also be found on a power table.

Figuring The Effect Size

• Raw Score Effect Size - calculated by taking the difference between the Population 1 mean and the Population 2 mean • Standardized Effect Size - calculated by dividing the raw score effect size for each study by each study's population standard deviation

Type I Error

• Rejecting the null hypothesis when the null hypothesis is true - You find an effect when in fact there is no effect. • A Type I error is a serious error as theories, research programs, treatment programs, and social programs are often based on conclusions of research studies. • The chance of making a Type I error is the same as the significance level. - If the significance level was set at p < .01, there is less than a 1% chance that you could have gotten your result if the null hypothesis was true. - To reduce the chance of making a Type I error, researchers can set a very stringent significance level (e.g., p < .001).

Hypothesis Testing When the Population Variance Is Unknown

• Restate the question about the research hypothesis and a null hypothesis about the populations. • Determine the characteristics of the comparison distribution. - population mean • This is the same as the known population mean. - population variance • Figure the estimated population variance. - S^2 = [∑(X - M)^2] / df • Figure the variance of the distribution of means. - S^2M = S^2/ N - standard deviation of the distribution of means • Figure the standard deviation of the distribution of means. - S^2M = √S^2M - shape of the comparison distribution • t distribution with N - 1 degrees of freedom • Determine the significance cutoff. - Decide the significance level and whether to use a one- or a two-tailed test. - Look up the appropriate cutoff in a t table. • Determine your sample's score on the comparison distribution. - t = (M - Population M) / SM • Decide whether to reject the null hypothesis. - Compare the t score of your sample and the cutoff score from the t table.

Steps for a t Test for Dependent Means

• Restate the question as a research hypothesis and a null hypothesis about the populations. • Determine the characteristics of the comparison distribution. - Make each person's two scores into a difference score. • Do all of the remaining steps using these difference scores. - Figure the mean of the difference scores. - Assume the mean of the distribution of means of difference scores = 0. - Find the standard deviation of the distribution of means of difference scores. • Figure the estimated population variance of difference scores. - S2 = [∑(X - M)2] / df • Figure the variance of the distribution of means of difference scores. - S2M = S2 / N • Figure the standard deviation of the distribution of means of difference scores. - S2M = √S2M - The shape is a t distribution with N - 1 degrees of freedom. • Determine the cutoff sample score on the comparison distribution at which the null hypothesis should be rejected. - Decide the significance level and whether to use a one- or a two-tailed test. - Look up the appropriate cutoff in a t table. • Determine the sample's score on the comparison distribution. - t = (M - Population M) / SM Decide whether to reject the null hypothesis. - Compare the t score for your sample to the cutoff score found using the t table.

t Tests in Research Articles

• Results from t tests are generally reported in the following format: - t (df) = x.xx, p < .05 • x.xx represents the t score. • Commonly, the significance level will be set at p < .05, but it is also often set at p < .01. • Research more commonly uses the t test for dependent means. - It is rare to see a study that uses a t test for a single sample. • Often a t test for dependent means will be given in the text, but sometimes results are reported in a table format.

Other Influences on Power

• Significance Level - Less extreme significance levels (e.g., p < .10) mean more power because the shaded rejection area of the lower curve is bigger and more of the area in the upper curve is shaded. - More extreme significance levels (e.g., p < .001) mean less power because the shaded region in the lower curve is smaller. • One- vs. Two-Tailed Tests - Using a two-tailed test makes it harder to get significance on any one tail. • Power is less with a two-tailed test than a one-tailed test.

Effect Size Conventions

• Standard rules about what to consider a small, medium, and large effect size - based on what is typical in behavioral and social science research • Cohen's effect size conventions for mean differences: - small effect size = .20 - medium effect size = .50 - large effect size = .80

Statistical Significance vs. Practical Significance

• Statistical Significance vs. Practical Significance - It is possible for a study with a small effect size to be significant. • Though the results are statistically significant , they may not have any practical significance. - e.g., if you tested a psychological treatment and your result is not big enough to make a difference that matters when treating patients • Evaluating the practical significance of study results is important when studying hypotheses that have practical implications. - e.g., whether a therapy treatment works, whether a particular math tutoring program actually helps to improve math skills, or whether sending mailing reminders increases the number of people who respond to the Census • With a small sample size, if a result is statistically significant, it is likely to be practically significant. • In a study with a large sample size, the effect size should also be considered.

The Power of Studies Using a t Test for Dependent Means

• Studies using a repeated-measures design (using difference scores) often have much larger effect sizes than studies using other research designs. - There is more power with this type of study than if the participants were divided into groups and each group was tested under each condition of the study. • The higher power is due to a smaller standard deviation that occurs in these type of studies. - The smaller variation is because you are comparing participants to themselves

Figuring Needed Sample Size for a Given Level of Power

• The main reason researchers consider power is to help them decide how many people to include in their studies. - Sample size has an important influence on power. - Researchers need to ensure that they have enough people in the study that they will be able see an effect if there is one.

Sample Size

• The more people there are in the study, the greater the power is. • The larger the sample size, the smaller the standard deviation of the distribution of means becomes. - The smaller the standard deviation of the distribution of means, the narrower the distribution of means—and the less overlap there is between distributions leading to higher power. • Remember that though sample size and effect size both influence power, they have nothing to do with each other.

Degrees of Freedom (df)

• The number by which you divide to get the estimated population variance • Number of scores free to vary when estimating a population parameter - If you know the mean of the population and all but one of the scores in the sample, you can figure out the score you don't know. • Once you know the mean, one of the scores in the sample is not free to have any possible value and the degrees of freedom then would = N - 1

The t Score

• The sample's mean score on the comparison distribution • It is calculated in the same way as a Z score, but it is used when the variance of the comparison distribution is estimated. • It is the sample's mean minus the population mean divided by the standard deviation of the distribution of means. t = M - Population M/ SM • If your sample's mean was 35, the population mean was 46, and the estimated standard deviation was 5, then the t score for this example would be -2.2. • This sample's mean is 2.2 standard deviations below the mean.

What Determines the Power of a Study?

• The statistical power of a study depends on: - how big an effect the research hypothesis predicts • effect size - how many participants are in the study • sample size - other factors that influence power include: • significance level chosen • whether a one-tailed or two-tailed test is used • the kind of hypothesis-testing procedure used

Using the t Table

• There is a different t distribution for any particular degrees of freedom. • The t table is a table of cutoff scores on the t distribution for various degrees of freedom, significance levels, and one- and two-tailed tests. • The t table only shows positive scores. • A portion of a t table might look like this: Chart on week 9 CHAPTER 8 section 14 slide 7

Deciding Whether to Reject the Null Hypothesis

• This is exactly the same as for the other hypothesis-testing procedures discussed in earlier chapters. - You will compare the t score for your sample to the cutoff score found using the t table to decide whether to reject the null hypothesis.

The t Distribution

• When the population variance is estimated, you have less true information and more room for error. - The shape of the comparison distribution will not be a normal curve; it will be a t distribution. • t distributions look like the normal curve—they are bell shaped, unimodal, and symmetrical—but there are more extreme scores in t distributions. - Their tails are higher. • There are many t distributions, the shapes of which vary according to the degrees of freedom used to calculate the distribution. - There is only one t distribution for any particular degrees of freedom.

Decision Errors

• When the right procedures lead to the wrong decisions • In spite of calculating everything correctly, conclusions drawn from hypothesis testing can still be incorrect. • This is possible because you are making decisions about populations based on information in samples. - Hypothesis testing is based on probability.

The Distribution of Differences Between Means

• When you have one score for each person with two different groups of people, you can compare the mean of one group to the mean of the other group. - The t test for independent means focuses on the difference between the means of the two groups. • The comparison distribution is a distribution of differences between means. - created by randomly selecting one mean from the distribution of means from the first group's population - randomly selecting one mean from the distribution of means for the second group's population - Subtract the mean from the second distribution of means from the mean from the first distribution of means to create a difference score between the two selected means. - Repeat this process a large number of times and you will have a distribution of differences between means. » Note that this is not the actual way a distribution of means is created, but conceptually this is what a distribution of means is.

Key Points

• When you have to estimate the population variance from scores in a sample, you will use a formula that divides the sum of square deviation scores by the degrees of freedom. • With an estimated population variance, the comparison distribution is a t distribution; it is close to normal, but varies depending on the associated degrees of freedom. • A t score is a sample's number of deviations from the mean of the comparison distribution this is used in situation when the population variance is estimated. • A t test for a single sample is used when the population mean is known but the population variance is unknown. • A researcher would use a t test for dependent means when there is more than one score for each participant. In this case you would use difference scores. • An assumption of the t test is that the population distribution is normal, but even if the distribution is not normal, the results are fairly accurate. • When testing hypotheses with t tests for dependent means, the mean of Population 2 is assumed to be 0. • effect size for t tests = mean of the difference scores/standard deviation of the difference scores • Power or sample size can be looked up using a power table. • The power with a repeated-measures design is usually much higher than that of most other designs with the same number of participants. • t tests for dependent means are often found in the text or in a table of a research article in this format: t (df) = x.xx, p < .05

Type II Error

• With a very extreme significance level, there is a greater probability that you will not reject the null hypothesis when the research hypothesis is actually true. - concluding that there is no effect when there is actually an effect • The probability of making a Type II error can be reduced by setting a very lenient significance level (e.g., p < .10).

Basic Principle of the t Test: Estimating the Population Variance from the Sample Scores

• You can estimate the variance of the population of individuals from the scores of people in your sample. - The variance of the scores from your sample will be slightly smaller than the variance of scores from the population. • Using the variance of the sample to estimate the variance of the population produces a biased estimate. • Unbiased Estimate - estimate of the population variance based on sample scores, which has been corrected so that it is equally likely to overestimate or underestimate the true population variance • The bias is corrected by dividing the sum of squared deviation by the sample size minus 1 - S2 = ∑(X - M)^2/ N - 1

Review of the Z test, t Test for a Single Sample, and t Test for Dependent Means

• Z Test - Population variance is known. - Population mean is known. - There is 1 score for each participant. - The comparison distribution is a Z distribution. - Formula Z = (M-Population M) / Population SDM - The best estimate of the population mean is the sample mean. • t Test for a Single Sample - Population variance is not known. - Population mean is known. - There is 1 score for each participant. - The comparison distribution is a t distribution. - df = N - 1 - Formula t = (M -Population M) / Population SM • t Test for Dependent Means - Population variance is not known. - Population mean is not known. - There are 2 scores for each participant. - The comparison distribution is a t distribution. - df = N - 1 - Formula t = (M -Population M) / Population SM