Homework 1.4
Briefly describe the eight guidelines for critically evaluating a statistical study. 1. For example, you should understand the goal of the study, the population that was under study, and whether the study was observational or an experiment. 2. In particular, look for any potential biases on the part of the researchers. 3. That is, decide whether the sampling method was likely to produce a representative sample. 4. Ambiguity in the variables can make it difficult to interpret reported results. 5. If the study neglected potential confounding variables, its results may not be valid. 6. In particular, look for anything that might tend to produce inaccurate or dishonest responses. 7. For example, check whether the study really supports the conclusions that are presented in the media. 8. For example, evaluate whether the study achieved its goals. If so, do the conclusions make sense and have practical significance?
1. Get a big picture view of the study. 2. Consider the source. 3. Look for bias in the sample. 4. Look for problems defining or measuring the variables of interest. 5. Beware of confounding variables. 6. Consider the setting and wording in surveys. 7. Check that results are presented fairly. 8. Stand back and consider the conclusions.
n author based conclusions about the general population of all women on 4500 replies that she received after mailing 100,000 questionnaires to various women's groups. Identify and explain a source of bias in this study. Then suggest how the bias might have been avoided.
Participation bias occurs because returning the questionnaires was voluntary. This bias might have been avoided by: - The author could have interviewed a randomly selected sample of women. - The author could have followed up with phone calls or other mailings.
Describe and contrast selection bias and participation bias in sampling. 1. Selection bias occurs whenever ___________________. 2. Participation bias occurs any time __________________.
1. researchers select their sample in a biased way 2. researchers select their sample in a biased way Selection bias occurs whenever researchers select their sample in a biased way that makes the sample unrepresentative of the population, such as a poll that only surveys members of a particular political party but draws conclusions about the entire population. Participation bias occurs any time participation in a study is voluntary, such as when surveys are mailed to respondents.
A clothing company sponsored a survey showing that dressing stylishly is critically important for success in a job interview. Determine which of the eight guidelines for evaluating statistical studies appears to be most relevant. Explain your reasoning.
Guideline 2 (Consider the source. In particular, look for any potential biases on the part of the researchers.) is most relevant because the company can benefit from the survey results.
An independent research firm conducts a survey of adults who view movies. Each interview subject is asked this question: "Is there is too much violence in movies?" Determine which of the eight guidelines for evaluating statistical studies appears to be most relevant. Explain your reasoning.
Guideline 4 (Look for problems defining or measuring the variables of interest. Ambiguity in the variables can make it difficult to interpret reported results) is most relevant because "too much" is not well defined and is difficult to measure.
Politicians commonly believe that they must make their political statements (often called sound-bytes) very short because the attention span of listeners is so short. A similar effect occurs in reporting statistical news. Major statistical studies are often reduced to one or two sentences. A summary of a statistical report, taken from a news source, is given below. Describe what crucial information is missing in the given statement below and what more should be known, before the report is acted on. A newspaper headline reported that "More companies try to bet on forecasting weather." The article gave examples of companies believing that long-range forecasts are reliable, and four companies were cited.
No information is given to justify the statement that "more" companies try to bet on weather forecasting in the headline. If the four cited companies are the only new ones using the forecasting, the increase is relatively insignificant.
What is peer review? How is it useful?
Peer review is having experts in a field evaluate a research report before it is published. This lends credibility because it implies that other experts agree that the study was carried out properly. The most common process by which scientists examine each other's research is called peer review. Reputable scientific journals require all research reports to be peer reviewed before the research is accepted for publication. Peer review does not guarantee that a study is valid, but it lends credibility because it implies that other experts agree that the study was carried out properly.
The headline "Drugs shown in 98 percent of movies" accompanied a news story that described a "government study" claiming that drug use, drinking, or smoking was depicted in 98% of the top movie rentals. Discuss whether the headline accurately represents the story.
The headline refers to drugs whereas the story refers to "drug use, drinking, or smoking." The headline is very misleading because the term "drugs" is generally considered to consist of drugs other than cigarettes or alcohol. Also, all movies consist of more than just the top movie rentals.
Decide whether the statement below makes sense or does not make sense. Explain your choice. A pollster plans to improve survey results by only conducting polls in which the margin of error is zero.
The statement does not make sense. A margin of error of zero implies that there is no uncertainty in a survey result. This could happen only if the entire population was surveyed, rather than just a sample.
Decide whether the following statement makes sense (or is clearly true) or does not make sense (or is clearly false). Explain your reasoning. The TV survey got more than 1 million phone-in responses, so it is clearly more valid than the survey by the professional pollsters, based on interview with only a few hundred people.
The statement does not make sense. The eight guidelines for evaluating a statistical study need to be reviewed before one study can be called more valid than another.