Research Methods Quiz 4

What two suggestions does the APA publication manual make regarding information to accompany a null hypothesis test?

1) Always include the effect size measure (like Cohen's d or Pearson's r), so that it is always clear how strong the relationship is. 2) Use confidence intervals instead, so you understand that most people fall within a particular range.

Discuss two criticisms that have been lodged against Null Hypothesis testing.

1) It is easy for researchers to misunderstand, especially what the p value means. This often leads to misguided optimism. 2) The logic of an arbitrary line of significance being drawn at .05.

Explain how a finding can be statistically significant but lack practical significance.

A finding can be statistically significant but otherwise basically irrelevant, such as if one thing is shown to be helpful but it is more expensive and time consuming than something else that works just as well.

Explain what meta-analysis is and why it is important.

A meta-analysis is taking all of the research on a topic and combining it. It is important because it can reveal the overall mean from all of the studies and address numerous issues and factors related to the idea.

Imagine that you conduct a t test and the p value is .02. How could you explain what this p value means to someone who is not already familiar with null hypothesis testing? Be sure to avoid the common misinterpretations of the p value.

A p value is the chance that we would get these results even if the relationship was not significant, i.e. due to chance/random. This low of a p value indicates it is unlikely that this relationship would be shown in the sample if there was no relationship in the population.

Define: Type I error, Type II error

A type one error is when the null hypothesis is true and you reject it anyway (is not significant but you say it is). A type two error is when the alternative hypothesis is true but you retain the null hypothesis (is significant but you say its not).

A study is conducted to see whether mindfulness training reduces test anxiety. Use plain English (no equations) to express...a...what is the null hypothesis for this study?

Any observed difference in test anxiety levels in the sample is due to chance and does not reflect a relationship between test anxiety and mindfulness training in the population.

Describe the steps in a basic meta-analysis.

First, you have to determine what topic to focus on. Then, you find all of the available research that addresses that question or topic, and find/compute the effect size for each. Then, you combine for an overall effect size and test potential moderators.

If a researcher rejects the null hypothesis, what exactly does this mean? [What has happened in the study? Use question 2 above as an example.]

If the null hypothesis is rejected, the sample's relationship reflects the population's relationship. The results were extreme enough to be under the alpha level, i.e., there is a slim enough chance that those results happened by chance that you can believe it is statistically significant.

Explain why some people say the field of psychology is experiencing a "replication crisis."

It can be difficult or impossible to replicate some studies, and sometimes even when it is done, the results don't support the original study's results.

What are the primary strengths and weaknesses of meta-analyses?

It can result in more definite answers and uncover areas that need more information. However, it can only be done when there are enough existing studies, it doesn't eliminate the publication bias, and if all of the studies are flawed, then the meta-analysis will be as well.

Explain publication bias and its implications for interpreting published research.

Publication bias is the fact that journals will be more likely to publish a study with a finding than a study that did not have their hypothesis supported. This leads to the file drawer problem, where "successful" studies are published but "non-successful" studies are stored away and not shared; so, if one study later finds the results to be significant, nobody is aware of the many others that found the results insignificant.

Describe some important open science practices that are becoming more common in psychology.

Revised standards and preregistration (which is describing your study in detail before it is even completed) are practices that are increasing in popularity.

What role does sampling error play in determining the presence of a statistical relationship?

Sampling error is when different randomly chosen samples can have different results. This means that any given sample may or may not indicate the presence of a real statistical relationship in the population.

What is "statistical power"? How is it related to Type II error?

Statistical power is the probability that you will reject the null hypothesis, considering the sample size and expected strength of the relationship. The higher the statistical power, the less likely it is to make a type two error.

If the alpha-level is changed from .05 to .01, what effect will this have on the likelihood of a) Type I error b) Type II error?

The likelihood of a type one error goes down, but type two error goes up.

A study is conducted to see whether mindfulness training reduces test anxiety. Use plain English (no equations) to express...a what is the alternate hypothesis for this study?

The observed difference in test anxiety levels is statistically significant, so we can reject the null hypothesis—the relationship in the sample is not yet disproven to be accurate for the population as well.

Define: p-value, alpha-level. What is the relationship between these terms?

The p value is the probability of getting a sample result that is the same as one if the null hypothesis were true; the alpha level is the criterion for deciding that a p value is low enough to reject the null hypothesis, usually .05. Their relationship is that the alpha level determines which p values are low enough to be significant.

How do sample size and relationship strength influence the results of a hypothesis test?

The stronger the sample relationship and the larger the sample size, the less likely that a result would be if the null hypothesis were true. Basically, if you have a big population, it's harder to believe that things are due to chance a larger sample better reflects the population and becomes more unlikely that things are just random based on the few people you picked. A stronger relationship also makes it less likely to be by chance, because if it's by chance it would likely be a relatively small relationship.

Explain what the open science movement is and list some of its goals.

This is the movement toward making studies more transparent to other psychologists and the public. It's goals are to be able to distinguish valid results more easily from flukes and encourage more exact replications, as well as encouraging non-finding studies to be more available.

Discussion: If you were a researcher, what concerns would you have about making your research materials and data publicly available online?

This would be a good thing because then somebody else could replicate the study, but I would be aware that somebody could disprove me and/or roast me for doing something incorrectly. Also, somebody could steal intellectual property, but I'm sure there are ways to prevent that.

List several reasons that published research results can be misleading or even wrong.

Type I and Type II errors do occur, and there is a publication bias. There is also researcher degree of freedom.