Homework 1.4

Briefly describe the eight guidelines for critically evaluating a statistical study. 1. For​ example, you should understand the goal of the​ study, the population that was under​ study, and whether the study was observational or an experiment. 2. In​ particular, look for any potential biases on the part of the researchers. 3. That​ is, decide whether the sampling method was likely to produce a representative sample. 4. Ambiguity in the variables can make it difficult to interpret reported results. 5. If the study neglected potential confounding​ variables, its results may not be valid. 6. In​ particular, look for anything that might tend to produce inaccurate or dishonest responses. 7. For​ example, check whether the study really supports the conclusions that are presented in the media. 8. For​ example, evaluate whether the study achieved its goals. If​ so, do the conclusions make sense and have practical​ significance?

1. Get a big picture view of the study. 2. Consider the source. 3. Look for bias in the sample. 4. Look for problems defining or measuring the variables of interest. 5. Beware of confounding variables. 6. Consider the setting and wording in surveys. 7. Check that results are presented fairly. 8. Stand back and consider the conclusions.

n author based conclusions about the general population of all women on 4500 replies that she received after mailing​ 100,000 questionnaires to various​ women's groups. Identify and explain a source of bias in this study. Then suggest how the bias might have been avoided.

Participation bias occurs because returning the questionnaires was voluntary. This bias might have been avoided by: - The author could have interviewed a randomly selected sample of women. - The author could have followed up with phone calls or other mailings.

Describe and contrast selection bias and participation bias in sampling. 1. Selection bias occurs whenever ___________________. 2. Participation bias occurs any time __________________.

1. researchers select their sample in a biased way 2. researchers select their sample in a biased way Selection bias occurs whenever researchers select their sample in a biased way that makes the sample unrepresentative of the​ population, such as a poll that only surveys members of a particular political party but draws conclusions about the entire population. Participation bias occurs any time participation in a study is​ voluntary, such as when surveys are mailed to respondents.

A clothing company sponsored a survey showing that dressing stylishly is critically important for success in a job interview. Determine which of the eight guidelines for evaluating statistical studies appears to be most relevant. Explain your reasoning.

Guideline 2 (Consider the source. In​ particular, look for any potential biases on the part of the researchers.) is most relevant because the company can benefit from the survey results.

An independent research firm conducts a survey of adults who view movies. Each interview subject is asked this​ question: "Is there is too much violence in​ movies?" Determine which of the eight guidelines for evaluating statistical studies appears to be most relevant. Explain your reasoning.

Guideline 4 (Look for problems defining or measuring the variables of interest. Ambiguity in the variables can make it difficult to interpret reported results) is most relevant because "too much" is not well defined and is difficult to measure.

Politicians commonly believe that they must make their political statements​ (often called​ sound-bytes) very short because the attention span of listeners is so short. A similar effect occurs in reporting statistical news. Major statistical studies are often reduced to one or two sentences. A summary of a statistical​ report, taken from a news​ source, is given below. Describe what crucial information is missing in the given statement below and what more should be​ known, before the report is acted on. A newspaper headline reported that​ "More companies try to bet on forecasting​ weather." The article gave examples of companies believing that​ long-range forecasts are​ reliable, and four companies were cited.

No information is given to justify the statement that​ "more" companies try to bet on weather forecasting in the headline. If the four cited companies are the only new ones using the​ forecasting, the increase is relatively insignificant.

What is peer​ review? How is it​ useful?

Peer review is having experts in a field evaluate a research report before it is published. This lends credibility because it implies that other experts agree that the study was carried out properly. The most common process by which scientists examine each​ other's research is called peer review. Reputable scientific journals require all research reports to be peer reviewed before the research is accepted for publication. Peer review does not guarantee that a study is​ valid, but it lends credibility because it implies that other experts agree that the study was carried out properly.

The headline​ "Drugs shown in 98 percent of​ movies" accompanied a news story that described a​ "government study" claiming that drug​ use, drinking, or smoking was depicted in​ 98% of the top movie rentals. Discuss whether the headline accurately represents the story.

The headline refers to drugs whereas the story refers to​ "drug use,​ drinking, or​ smoking." The headline is very misleading because the term​ "drugs" is generally considered to consist of drugs other than cigarettes or alcohol.​ Also, all movies consist of more than just the top movie rentals.

Decide whether the statement below makes sense or does not make sense. Explain your choice. A pollster plans to improve survey results by only conducting polls in which the margin of error is zero.

The statement does not make sense. A margin of error of zero implies that there is no uncertainty in a survey result. This could happen only if the entire population was​ surveyed, rather than just a sample.

Decide whether the following statement makes sense​ (or is clearly​ true) or does not make sense​ (or is clearly​ false). Explain your reasoning. The TV survey got more than 1 million​ phone-in responses, so it is clearly more valid than the survey by the professional​ pollsters, based on interview with only a few hundred people.

The statement does not make sense. The eight guidelines for evaluating a statistical study need to be reviewed before one study can be called more valid than another.