TEXTBOOK: Ch. 2: Scientific Methods in Media Effects Research

*cross-sectional survey* *longitudinal survey* *trend study* *cohort study* *age cohort* *panel study*

Types of Surveys: The survey conducted by Ahern and her colleagues on TV exposure to the events of 9-11 is one of the most common types of survey reported in social science research. It was a survey of a single sample of a population. This kind of survey is commonly referred to as a *__________________________*. The distinguishing characteristic of a cross-sectional survey is that it occurs at a single point in time and involves a single sample. Often, researchers are not satisfied with a single sample. They have research questions that cannot be answered unless a separate questionnaire is administered at more than 1 point in time. When a researcher designs a survey project that includes more than 1 sample taken at different points in time, the survey design is referred to as longitudinal. There are 3 kinds of *_____________________* designs. The first type of longitudinal design is a *______________*. About 8 years ago, I became especially interested in the possible changes that were taking place among college students in their use of new technology. I decided to begin surveying a small group from each freshman class. Before the project ended, I had data from the freshman classes of 2006 through 2009. You probably won't be surprised to learn that over those 4 years, the data suggest that students spent more time each year engaged in text messaging and Facebook. Notice that in a trend study, the individuals who are surveyed the first time are not the same individuals who are surveyed the second time. The only similarity between the 4 groups I surveyed is that everyone was a member of a freshman class (but a different freshman class) at the time of the survey. A second type of longitudinal survey is a *_________________*. Suppose I became interested in any change in the use of new technology among students who were in the freshman class of 2006. Instead of continuing to survey freshmen from each year's entering class, I might choose another sample of individuals who were sophomores in 2007, juniors in 2008, or seniors in 2009. Samples of students from these classes would be members of the same *______________* as the freshmen from the entering class of 2006. But they wouldn't be the same students. Perhaps you have seen news reports on how the attitudes of the post-World War II baby boomers have changed as they have grown older. These changes are typically detected with a cohort design. If people between the ages of 40 and 50 were surveyed in 2005, then people between the ages of 50 and 60 would be surveyed in 2015. The cohort survey allows the researcher to assess changes in a general cohort group—but not in the same individuals. The final type of longitudinal survey design is a *________________*. If I wanted to see how use of new technology changed in exactly the same individuals, then I would use a panel design. Long-term panel designs are relatively rare because of the time and effort involved in keeping track of the same individuals over time. In Chapter 5 on media violence, you will learn about a long-term panel study that followed young children into their adult years. Although such studies are rare in mass-communication research, the results are usually very important in revealing trends that no other method is capable of detecting. Short-term panel designs are used more frequently. Researchers often design short-term panels during political campaigns to track attitudinal changes that might take place in response to media messages and other political events.

*unit of analysis*

Units of Analysis: After the sample is selected, the researchers must decide what units of the content will be coded. For example, in a newspaper article, the researcher might code each paragraph for certain attributes. Thus, the paragraph would be the *__________________*. If each sentence were coded, then the sentence would be the unit of analysis. Sometimes TV shows are broken into scenes and each scene is coded. In this case, although the authors don't clearly state the unit of analysis, it appears to be the entire novel and each novel could have had multiple instances of alcohol or substance use that were coded individually.

*mood-management theory*

An Example: Using a Video Game to Repair a Bad Mood: One topic that some media researchers have shown interest in over the years is how our media use affects our emotional moods. In fact, a researcher named Dolf Zillmann has proposed an entire theory on this topic known as *____________________*. The theory states that people deliberately use media to alter their moods. For example, suppose you were in a bad mood and you had the chance to play a video game. Would playing the game help to alleviate your negative mood? Recently, Nicholas Bowman and Ron Tamborini attempted to answer this question by conducting an experiment to see whether a person's negative mood could be repaired in a more positive direction as a result of playing with a flight simulator video game. According to the theory, 1 reason people turn to media to repair a negative mood is because of the "intervention potential" of media use. Put simply, using media can distract a person from their negative mood and cause them to feel better. If this logic is correct, the more involved a person becomes with media, the more their negative mood should dissipate.

*Content analysis*

ANALYZING MEDIA CONTENT: In general, the first step in scientific investigation is to "describe" the phenomenon of interest with precision. *____________________* allows a researcher to describe the nature of the content of communication in a systematic and rigorous fashion. Content analysis can be applied to almost any type of communication, but it is particularly appropriate for mass media messages because it permits us to describe precisely a vast diversity of message content that might otherwise prove elusive. Content analysis is a logical beginning point for the investigation of media effects because it helps us to discover what content is present that might be bringing about various effects. The controversy about the impact of media violence on children would hardly capture our attention if none of the programs or movies that were made contained any violent scenes. On the other hand, it is important to understand that *the results of a content analysis do not permit one to make inferences about the effects of that content.* I used to carry a card in my wallet that was distributed by a group called TV Tune-In, located in Cleveland, Ohio. Over the years, the card became faded and worn so I retired it to a desk drawer for posterity. One of the "TV Awareness Facts" that appears on the card is particularly striking: "More guns are fired on TV in one evening than are fired in the course of an entire year by a metropolitan police force of 504 officers!" This "fact" was based on content analyses of prime-time TV and since the card is at least 30 years old, the statistics it reports can no longer be trusted as dependable. They'd have to be updated by doing new content analyses. The researchers who conducted the study that this old statistic is based on arrived at this figure after counting the number of guns fired during an average evening of prime-time offerings. It may seem like an easy jump to move from this "fact" to the inference that TV influences people to shoot guns. But such an inference would be unjustified. It could be the case that people become disgusted with all the guns fired on TV and consequently use guns less frequently than they would otherwise. Alternatively, it could be that people carefully separate the world of TV and the real world such that seeing guns fired on TV has no effect at all on guns fired in real life. The facts themselves that come from a content analysis don't ever permit us to answer the question about the "effects" of the content. To answer that question, other research methods must be used. Before we examine some of those other methods, let's take a closer look at the method of content analysis and examine the role that it can play in mass-communication research.

*Replication* *Convergence*

CONTROVERSY ABOUT RESEARCH METHODS: This chapter has introduced you to the research methods used in media effects studies. As you read about each of the methods, you may have wondered about the extent to which we should rely on them for our understanding of the effects of the media. For example, perhaps it occurred to you that when people are asked questions on a survey, they might not respond with truthful answers. Or maybe you wondered about whether a carefully controlled laboratory experiment can really tell us very much about behavior that happens in more natural environments—are laboratory experiments too artificial? For years, researchers have debated important questions like these. A full consideration of each potential problem of a particular research method is beyond the scope of this book. Study Box 2-2 provides a brief overview of 4 types of experimental validity that you can use as an initial checklist for evaluating the validity of any reported findings from an experiment. In spite of the difficulties with individual research methods, media effects researchers gain confidence in the results of research studies by examining the same research problem in a number of different ways. If a particular finding seems to show up again and again in studies of different types, scientists would refer to this as replicating a research result with convergent evidence. *__________________* refers to the fact that the same result can be observed over and over again. *_________________* refers to the fact that the use of different methods still leads to the same general conclusion. In the end, researchers are reluctant to declare that they know anything about media effects until they have results from more than 1 or 2 studies. This caution helps to protect against accepting research findings that could be the product of unseen biases that can creep into a researcher's work and affect everything from the experimental design to the statistical analyses. Recently, one researcher attempted to assess these sorts of biases and rendered the very discouraging conclusion that, "most claimed research findings are false." If that's true, then replication and convergence are critical parts of the research process. If a finding can't be replicated or only emerges using one type of method, it signals that caution is warranted and it's probably premature to accept the finding as true. Although no single research method is perfect or beyond criticism, careful application of the 3 methods discussed in this chapter provides a powerful arsenal of tools for learning about the ways that media messages affect people.

*category scheme*

Categories: In every content analysis, the content of the message is coded according to a *____________________*. Coyne and her coauthors wanted to code each incident in several ways. For example, they wanted to code type of substance used (e.g., alcohol, tobacco, illegal drugs, prescription drugs, etc.) as well as the reason for its use (e.g., celebrating, relaxing, coping, addiction, etc.), the age of the user (minor or adult), and the consequences of use (no consequences, positive consequences, or negative consequences). Sometimes, it is difficult to tell from a research report exactly how the category scheme was actually applied. For example, suppose a character is using alcohol at one point in the novel and the scene changes for the next 10 pages and then returns to the same instance of alcohol use. Does this count as 2 instances of alcohol use or is it just counted as a single incident? The only way to get answers to questions like this is to carefully examine the coding rules that the authors applied in the study. In some studies, the coding rules aren't completely specified. In cases where they aren't, the only way to discover the rules is to try to contact the authors directly and ask them for more detail. In this case, Coyne and her coauthors recognized that readers might have numerous questions about the coding rules and that the journal publishing their article had placed them under strict space constraints. Consequently, they invited readers to contact them with any questions.

*coding reliability*

Coding Agreement: After more than one coder has coded the data, some statistical index of agreement among the coders is computed. Sometimes this statistic is simply calculated by dividing the number of times the coders agree by the total number of coding decisions they make. For example, consider the case of a researcher who wanted to code the biological sex of the person who consumed alcohol (Coyne and her coauthors did code for this information). Suppose that there were 10 incidents of alcohol use and that 2 independent coders agreed on the sex of the user in 9 out of the 10. Their percentage of agreement would be 90% or .90. Although this may seem like a pretty good percentage of agreement, consider the fact that there are only 2 possible categories in this case: male and female. The possibility of agreeing just by chance is 50%. Because of the possibility of chance agreements, many researchers compute coding agreement (often referred to as *___________________*) with a special formula that makes an adjustment for the chance hits. A rule of thumb in content analysis is that coding reliability must be at least .70 (a coefficient of 1.0 indicates perfect agreement and lower levels are accepted if a formula that adjusts for chance agreements is employed). In the study of popular adolescent novels, the authors used 5 independent coders who had been trained for 3 weeks to code substance use in the novels. For purposes of demonstrating reliability, all 5 coders coded the content of the same 4 books (10% of the total sample) and their coding reliability was never lower than .66 for any of the categories that they coded. Normally, scholars would like to see the coding reliability statistic for each separate category that was coded, but this information is not always reported. Since Coyne and her coauthors didn't provide this information in the published article, a reader could always try to contact the authors directly for this information. That's what I did in preparation for writing this section. In a simple email exchange that took just a matter of hours (Coyne and I didn't know each other before our email exchange), Coyne provided the coding reliability for each of the categories that she used. As it turned out, her coders had perfect coding reliability (1.0) for all but 3 of the categories used in the study. The coders had less than perfect agreement when they coded whether or not there was an instance of substance use (.66), the age of the user (.71), and the consequences associated with the use (.71). Even though agreement wasn't perfect for these categories, it was high enough to trust the coders as they independently coded the remaining novels. After coding reliability is demonstrated on a subset of the entire sample, the coders usually go on to divide the labor, with each coder taking a portion of the remaining material to be coded.

*control group*

Control Groups: To properly assess the impact of a manipulated variable in an experiment, some sort of *____________________* is typically used. Suppose that, in the experiment on sugar substitutes, animals in the large-dose group were more likely to develop cancer than animals in the small-dose group. Such a finding might be interpreted to mean that the sugar substitute is dangerous and consumption should be avoided. Notice, however, that the experiment described earlier did not contain a control group that received no sugar substitutes. The inclusion of a control group in the experiment might completely change the interpretation of the results. What if, in an experiment that included a no-substitute control group, animals in the control group had the highest incidence of cancer? In this case, the interpretation of the results is completely changed. With the inclusion of the control group, it now appears that consumption of any level of the sugar substitute reduces the incidence of cancer. Heavy consumption reduces it less than light consumption, but both doses reduce it. These results are completely hypothetical, but they help to illustrate how the inclusion of a control group in an experimental design can often aid the researcher in interpreting the results of an experiment clearly. Before examining a specific example of an experiment designed to investigate a media effect, let's look briefly at some of the possible designs that a researcher might employ.

*correlation coefficient* *positive correlation* *negative correlation*

Correlation Coefficients: When researchers want to know whether 2 variables are related to each other, they often apply a statistical formula to the data and compute a *_________________________*. The particular formula for the correlation coefficient is commonly available in introductory statistics texts, and you don't need to be concerned with formulas here. But because of their importance in documenting empirical relationships between variables, it is helpful to know how to interpret correlation coefficients. Correlation coefficients are computed between 2 variables. The data in the obesity studies conform nicely to the sort of situation where correlation can be useful. The researchers who published these studies had a measure of 2 different variables for each person in the sample. If increases in 1 of the variables tended to go along with increases in the other variable, then the 2 variables would have a *____________________*. This was actually the case in the 2 obesity studies mentioned earlier. People who had higher TV viewing times tended to weigh more. Sometimes, increases in 1 of the variables tend to go along with decreases in the other variable. When this situation arises, the 2 variables have a *____________________*. Some studies have documented a negative correlation between income level and TV viewing. People who make more money tend to watch less TV. Of course, it is not always the case that the 2 variables being correlated are related to each other in either a positive or a negative way. In some cases, the 2 variables might be unrelated.

*correlation*

Criteria for Causal Relationships: The first thing a researcher needs to establish before making a claim that 1 thing causes another is that the 2 things in question are empirically related to each other. Recall the studies mentioned earlier that investigated the relationship between TV viewing and obesity. To document that watching TV "caused" people to become more obese, researchers first needed to establish that these 2 variables (TV viewing time and level of obesity) were actually related to each other. As it has turned out in several studies, the 2 variables are empirically related. That is, there was a tendency for the people who spent more time watching TV to also be the people with higher body weights. Researchers need a specific tool to detect an empirical relationship like this one. They can't simply eyeball the data and make a casual declaration that 2 variables seem to be either related or unrelated. The methods of science are more precise than that. The most common statistical tool used to determine relationships between 2 variables is the technique of *_________________*.

*pre-test, post-test* *pre-test, post-test, control group* *post-test-only*

Different Experimental Designs: Not every experiment is designed in exactly the same way. Researchers have various choices to make about how they want to set up a study and what they want to measure at which point in the experimental process. The study of experimental designs can get quite complicated; graduate students frequently take semester-long courses in experimental design. Although that sort of in-depth treatment is beyond the scope of this text, it might be useful to get at least some passing insight into the types of choices that experimenters have to make. Consider the case of a researcher who wants to know whether playing certain types of video games causes a person's heart rate to increase. There are at least 2 possible ways to set up the experiment. In the first approach, the researcher could have people sit quietly before playing one of several different video games in order to get a baseline heart rate reading. Then, immediately after playing a particular game, heart rate could be measured again. By comparing the first reading with the second across groups of people who were randomly assigned to play different games, the researcher could determine which game caused the greatest increase in heart rate. In this case, because the researcher measures heart rate both before and after playing the game (the game is the experimental stimulus), the experimental design would be called a *________________________* design. If the research added a control group condition in which people played no video game between the 2 heart rate measurements, it would be a *__________________________* design. In studies where heart rate is measured, the pre-test, post-test design would be a common approach. A second approach to this experiment is called the *_________________* design. You can probably guess what the difference in this design is just by thinking about its name. In the post-test-only design, the researcher would examine heart rate only once, after the game playing is over. What is the logic of taking this approach? Wouldn't the researcher be losing some valuable information by eliminating the first heart rate readings? Perhaps not. If you think about a carefully designed experiment, once the participants have been randomly assigned to the experimental conditions, the groups of people assigned to play different video games should have, on average, heart rates that are statistically equivalent to those of the other group. Recall that random assignment to conditions should theoretically result in groups that are equivalent on any variable of interest. If the heart rates of the different groups are theoretically equivalent at the start of the experiment, a researcher might not care very much about taking a pre-test measurement. It is true, however, that some information is lost with this type of design. In this case, the researcher wouldn't be able to describe how many average beats per minute a group either increased or decreased as a result of playing the game. But the researcher would still be able to make the crucial comparison that the experiment was intended to make. That is, the researcher would still be able to describe the differences in heart rate between the various groups right after the game playing was over. Is there any reason that a researcher would deliberately want to pass up the pre- test measure? The answer to this question is yes. Consider the situation where the pre-test measure might be a questionnaire about your attitude toward particular video games. Now imagine that after playing a game, you are asked to complete the same attitude measure that you filled out just before the beginning of the experiment. Instead of reporting your true attitude afterward, you might think about your responses on the first attitude measure and strive to be consistent with your earlier answers. In this case, the pre-test would have sensitized you to the post-test. If that sort of sensitization happens, the whole purpose of the experiment is defeated. Instead of being able to see how the experimental manipulation affected your attitude, the researcher would really be observing how the pre-test measure of attitude affected your response on the post-test. Researchers are often willing to lose a little pre-test information to avoid the risk of ruining the whole experiment. You can probably see from this discussion that designing good experiments requires plenty of careful thought and analysis. It usually takes several years of training before a researcher is able to consistently make good choices about experimental designs.

*manipulation of the variable*

Identical Treatment Except for the Manipulation: Random assignment to experimental conditions theoretically makes the experimental groups equivalent at the beginning of the experiment. Random assignment must be followed by identical treatment in all ways except for the *_____________________________*. In the sugar substitute experiment, the animals in all 3 conditions should have identical diets except for the difference in the dose of sugar substitute. They should be housed in identical cages with equal amounts of light and dark, equal numbers of other animals in their cages, and so on. In short, anything that introduces a difference in the experimental conditions, except the manipulated variable, must be avoided. Otherwise, once the experiment is over, the researchers can't be confident that differences in the manipulated variable among the experimental groups caused the dependent variable of interest (in this case, incidence of cancer). Recall the discussion of the 3 criteria for establishing causal relationships. Survey designs are capable of meeting 2 of the 3: revealing the presence of a relationship and establishing the time-order of the variables. As we saw, however, the survey design could not eliminate all third variables that might present themselves as possible alternative causes to consider in a given relationship. The experimental method, however, is able to meet all 3 criteria at once. Consider the experiment on sugar substitutes. If one of the conditions shows a higher incidence of cancer, then a relationship between consumption of the substitute and cancer has been found. That's the first criterion. If the experimental groups were equivalent at the beginning of the experiment, then time-order has also been established. The increased incidence of cancer in one of the groups occurred after consumption was manipulated. That's the second criterion. Finally, if the experimental groups were equivalent at the beginning of the experiment and were treated equally throughout the experiment except for the manipulation, then there are no third variables that can offer a rival explanation for how differences in cancer emerged in one of the groups. The experiment theoretically controls all possible third variables that might compete with the manipulation to explain the outcome of the experiment. It is this feature of the experiment that makes it such a valuable tool for investigating questions of media effects.

*statistically significant* *time-order* *third-variable explanations*

Interpreting Correlation Coefficients: When researchers compute correlation coefficients between 2 variables, the result is a specific number that provides an index of how strongly the variables are related. The formula for the correlation coefficient is designed so that variables that are perfectly related have an index of +1.0 (if the relationship is positive) or -1.0 (if the relationship is negative). No correlation coefficient can exceed +1.0 or -1.0. The smallest correlation that could exist between 2 variables is no relationship at all—indicated by a coefficient of zero (0.0). The statistical symbol for the coefficient is r. Typically, then, when correlation coefficients are reported, you might read something like r = .34 (if the relationship between the variables is positive) or r = -.21 (if the relationship between the variables is negative). Once a correlation coefficient has been computed (usually by a computer), its meaning must be interpreted. Suppose the correlation between 2 variables is indicated by r = .45. What do we know? Although we know that the 2 variables have some association, we still don't know how likely it is that this result occurred by chance alone. If the correlation between 2 variables is very likely to occur by chance, then we wouldn't want to attach too much importance to the particular coefficient that was computed. Fortunately, statisticians have determined how likely it is that correlation coefficients of given magnitudes occur by chance, given samples of various sizes. These probabilities appear in statistical tables and are built right into the computer programs that compute correlation coefficients. Scientists have agreed by consensus to adopt a particular standard for determining a chance occurrence. If a statistical result could happen by chance more than 5 times in 100, it is generally considered to be a chance finding. If, on the other hand, a statistical result could happen by chance 5 times in 100 or less, the result is considered unlikely to be due to chance. A statistical result that is unlikely to be due to chance is referred to as a *____________________* result. When correlation coefficients are reported in the scientific literature, they usually look something like this: r = .32, p < .04. The first part of this expression reports how likely it is that a correlation this large would occur by chance alone. In this case, it would be expected by chance less than 4 times in 100 and thus would be considered statistically significant. Although we're talking here about correlation coefficients, the concept of statistical significance is the same regardless of what type of statistical test is being applied. Recall the results from the content analysis on adolescent novels mentioned earlier. In 83% of the substance-use incidents that were coded, no negative or positive consequences were evident. Only 17% of the incidents included any description of positive or negative consequences. The difference in these 2 percentages was statistically significant. This means that when Coyne and her colleagues compared the 2 statistics with the appropriate test, the results indicated that a difference this large would only occur by chance less than 5 times in 100. In this case, the concept of statistical significance was applied to a difference between 2 percentages—not a correlation coefficient. Once a researcher determines that a correlation is statistically significant (i.e., unlikely to be due to chance), the first criterion for documenting a causal relationship has been met (see Study Box 2-1 for more detail on interpreting correlation coefficients). The researchers who documented that there was a significant positive correlation between TV viewing and obesity met this first criterion. But recall that there are 3 criteria for establishing causal relationships. Unless all 3 are met, the researcher is on faulty ground in declaring a causal relationship between the 2 variables. Determining that TV viewing and obesity are positively correlated is only the first step. In the study that showed a relationship between exposure to TV coverage about 9-11 and the tendency to score high on a measure of post-traumatic stress, it might have been tempting to conclude that exposure to TV actually caused post-traumatic stress. But the authors of that study correctly noted that they had only met the first step in establishing a causal relationship. There were other possible interpretations for their data. The second criterion that must be met to establish a causal relationship is the *_________________* of the 2 variables. One thing can't cause another thing unless it precedes it in time. A baseball soaring over the center-field fence cannot be the cause of the batter's swing that resulted in the ball's flight. To clearly establish causality, researchers must document the fact that *the variable doing the causing precedes the variable that is caused*. This second criterion often is more challenging than it first appears. Think about the surveys that documented an empirical relationship between TV viewing and obesity. We learned that people who watch a lot of TV tend to have higher body weights. But do we know from the survey data that TV viewing preceded body weight in the sequence of time? The answer to this question is clearly no. Because all the data in these surveys were collected at the same point in time using a cross-sectional survey design, the researchers have no way of sorting out which variable came first. Could body weight cause TV viewing instead of the reverse? If you think about it, that possibility is perfectly plausible. Perhaps, for whatever reasons, some people weigh more than others. Perhaps these heavier people prefer to lead a more passive lifestyle because exercise is an extra effort that strains the body. In their passivity, they might naturally turn to TV as a form of entertainment more frequently than to physical exercise. The same sort of possibility exists in the study on TV exposure to the events of 9-11 and post-traumatic stress. Perhaps people who were already high on post- traumatic stress were especially likely to be drawn to the TV coverage of a traumatic event. Some scholars have suggested that TV coverage of disasters may actually help people cope with a stressful event by emphasizing that the disaster is being addressed or controlled. It seems plausible then that people who are already high on a measure of stress might actively seek TV coverage of disasters to seek reassurance that everything will be fine. n this case, a cross-sectional survey is simply an inadequate tool for establishing the time-order between the 2 variables. One of the advantages of longitudinal surveys is that they can establish the time- order between variables. In Chapter 5 on media violence, you will read about a panel study that found that early viewing of TV violence was significantly correlated with aggressive behavior much later in life. Longitudinal surveys of this type are capable of establishing empirical relationships between 2 variables as well as establishing the time-order between the variables. For the sake of our example, let's suppose that the researchers studying TV viewing and obesity had conducted a longitudinal survey and had established that reports of TV viewing in the first survey were significantly correlated with weight gains reported in the second survey. In such a case, both of the first 2 criteria for establishing causal relationships would have been met. But this is still not enough evidence to establish the causal claim. There is a final criterion that also needs to be met. After empirical relationships and time-order have been established, a researcher must also establish that the observed relationship is not due to some unmeasured variable (sometimes called a "3rd" variable) that is causally related to both of the others. All possible *_____________________________* must be eliminated. In the studies on TV viewing and obesity, one plausible 3rd-variable explanation for the relationship might be socioeconomic status. People with low incomes may only be able to afford housing in poor, urban areas. Such areas may not provide many opportunities for recreation and physical exercise. People living in these neighborhoods may perceive that they are in danger when they go outside, so they prefer to stay indoors for their recreation, thus burning fewer calories and gaining more weight. TV viewing is the predominant type of entertainment available indoors, so these folks also watch more TV. In this scenario, low income would be causing both TV viewing and obesity. If a researcher examined TV viewing and obesity without including a measure of income, the empirical relationship might emerge and lead the researcher to think that watching TV causes obesity—but this would be a mistaken conclusion. The true cause of the relationship in this case (low income) would remain hidden. From the preceding discussion, it might seem impossible to design a study that simultaneously meets all 3 criteria for establishing a causal relationship. If an unmeasured variable might be responsible for the relationship between 2 measured variables, then clearly a sample survey (whether cross-sectional or longitudinal) can never provide the definitive evidence needed for a causal relationship. Because no survey can measure every possible variable, there is always a chance that a variable that went unmeasured is precisely the one that explains why 2 measured variables are related to each other. A good survey design can anticipate some of the obvious variables that might play the role of an explanatory third variable. Once these variables are measured, they can be tested to see if they might be responsible for an observed relationship between 2 other variables of interest (this was done in Tucker and Bagwell's study on TV and obesity). But no matter how clever the survey design, no survey can measure everything. Consequently, the survey is, in principle, not capable of satisfying all 3 of the criteria needed to say that 1 thing causes another. Where does this leave us? If the sample survey is capable of establishing only 2 of the 3 criteria for causal relationships (empirical relationship and time- order), is there an alternative method that might allow us to meet all 3 criteria simultaneously? The answer to this question leads to the last of the 3 fundamental methods used to study media effects: the experiment.

*dependent variable*

Manipulation of a Key Variable: Once an experimenter has identified a variable (usually called the independent variable) that is thought to be a potential cause of another variable (usually called the *_____________________*), the strategy is to manipulate the independent variable to create more than one experimental condition. The purpose of this manipulation is to be able to observe the impact of various levels of the independent variable on some dependent variable. My daughter has always been concerned about drinking too much diet soda. Her concern stems from reports she has heard from her friends that too much of the sugar substitute in diet soda might cause cancer. We did a little investigating and discovered that researchers have completed many experiments on this topic. In this case, the sugar substitute is the potential causal variable. Consequently, researchers manipulated this variable and created a number of experimental conditions. In one condition of a typical experiment, laboratory animals might be injected with very large doses of the substance over a period of weeks. In a second condition, moderate doses might be administered. A third group of animals might get very light doses. The animals in each of these conditions can then be compared for their incidence of cancer.

*epidemiological approach* *meta-analysis*

OTHER METHODOLOGICAL APPROACHES: Although content analyses, surveys, and experiments are the major tools of media effects research, 2 other useful tools are worth mentioning. The first of these is the *_________________________*. This approach is useful for studying the potential impact of media in the natural world. Epidemiological research is often conducted in the medical arena because it is not ethical to do experiments that might cause severe harm. The best example of this might be research on the effects of smoking on the incidence of cancer. Although researchers would never want to set up a study in which some people were randomly assigned to smoke large numbers of cigarettes, they can still study the potential causal link between smoking and cancer by simply observing what happens to people who smoke compared to those who don't. Essentially, then, the epidemiological approach is an observational science. Unlike the experiment, it doesn't seek to manipulate variables. Instead, it seeks to find connections between variables by simply observing what is happening in the world outside the laboratory. As you might suspect, because epidemiological researchers can't control variables the way experiments do in the laboratory, the interpretation of results must be done very carefully. There are many potential "third variables" that might account for a relationship found in an epidemiological study. For example, maybe smokers also tend to have different diets than non-smokers. Of course, researchers who take the epidemiological approach seek to make simultaneous observations on other variables so that they can rule out these types of explanations for a relationship. When some medical researchers who conduct experiments on laboratory animals discover that tobacco products cause cancer, and others who take the epidemiological approach discover that smokers are more likely to die of cancer than people who don't smoke, the combination of this converging evidence is more convincing than either type of evidence alone. Similarly, as you will discover in Chapter 5, epidemiological studies have revealed that violence in different societies tends to increase with the introduction of television. When this evidence is combined with evidence from carefully controlled laboratory experiments on the effects of media violence on aggression, we get a much more complete and compelling picture of how media violence might actually cause aggression. A second useful methodological tool is *_____________________*. Unlike the other methods discussed in this chapter, the meta-analysis is a technique that doesn't involve making observations on a sample of people. Instead, this technique relies on studies that have already been completed and uses those studies as the data for a new, overall summary. The key to meta-analysis is the way in which this summary of the existing studies is constructed. The summary is based on a very precise examination of the statistical effects reported in each individual study. The result of this examination is a statistical average over the entire set of studies that is taken to be a more reliable indication of any effect than the results of a single study alone. Of course, just as with any other methodology, certain problems and limitations arise. Some published studies don't contain the statistical details that are needed for the computations. In those cases, if the authors cannot be contacted to provide the missing information, there is no recourse but to drop the study from the overall analysis. Of course, the research community would be in deep trouble if everyone decided to simply do meta-analyses for the rest of their careers. Unless basic surveys and experiments are conducted and reported, the meta-analyst would have nothing to analyze. Still, it is important to recognize that this technique is very valuable for summarizing across a large number of studies and arriving at a conclusion that might not be evident from simply reading over the literature.

*random assignment*

Random Assignment to Experimental Conditions: One crucial feature in experiments is *________________________* of people (or animals) to experimental conditions. In experiments on the impact of sugar substitutes, laboratory animals are randomly assigned to one of the doses. Why is random assignment so important? The answer to this question should be clear upon thinking about the population of the laboratory animals. Suppose that one of the cages where a group of the animals is housed resides in a corner of the building that contains cancer- causing asbestos. If all the animals in that cage were placed in the same experimental condition, the researchers might erroneously conclude that the experimental treatment was responsible for a higher incidence of cancer in this condition. Or suppose that some of the animals are already developing the early stages of cancer. If those animals are spread evenly across the experimental groups, then researchers can still detect the impact of the sugar substitute. Random assignment to experimental conditions theoretically makes the experimental groups equivalent just prior to their exposure to the experimental manipulation. Consequently, any difference that emerges after the manipulation can be confidently attributed to the manipulation itself.

*coefficient of determination* *independent variable* *statistical importance*

STUDY BOX 2-1 Interpreting Magnitudes of Correlation Coefficients: Correlation coefficients can range in value from -1.0 to +1.0. Coefficients are treated as if they were zero (indicating no relationship between the variables), unless the statistical test for significance reveals that the magnitude of the correlation is not likely due to chance. It is important to keep in mind that even in the case of a zero correlation coefficient the variables still could be related in a non-linear fashion. That is, correlation coefficients test only for linear relationships—not for relationships that are non-linear. Imagine a situation where people who watch no TV at all are very high on knowledge of current events, perhaps from reading a lot. In the same scenario, people who watch TV for many hours are also well informed, in this case because they watch many news programs. A third group, with the lowest knowledge of current events, consists of moderate watchers of TV. Although this example is completely hypothetical, it does illustrate a non-linear relationship between TV viewing and knowledge of current events. In such a case, the correlation coefficient would be a poor test of a relationship between the 2 variables. Fortunately, researchers are able to apply special statistics to cases where variables are related in non-linear ways. When a statistical test does indicate that a correlation is not likely to be the result of chance (p < .05), then it is important to examine the magnitude of the relationship. One statistician, J. P. Guilford, suggested that correlation coefficients between 0.0 and .20 were very slight and should be viewed as indicating nearly no relationship at all. Coefficients between .20 and .40 should be considered as indicating small but definite relationships. Moderate relationships are indicated by coefficients between .40 and .70. Coefficients above .70 (rare in the social sciences) are considered to be substantial and large. Another way of judging the size of the relationship between 2 variables is to simply square (r2) the value of the correlation coefficient. This value, the *_________________________*, provides a statistical indication of how much information or variance in the dependent variable is explained or accounted for by knowing the values of the *_______________________*. For example, if the correlation between TV viewing and obesity was r = .30, p < .05 you would know that the relationship was not likely due to chance. In addition, you would know that TV viewing was able to account for 9% (.302 = .09) of the variance in the measure of obesity. In this example, 91% of the variance in obesity would be unexplained by TV viewing. This is a valuable tool to apply because it tells us something about the relative *_______________________* of the relationship between the 2 variables.

*experimental validity* *"Statistical Validity"* *"Internal Validity"* *"External Validity"* *"Construct Validity"*

STUDY BOX 2-2 Evaluating the Validity of Any Experiment: Whether an experiment is conducted to determine the effects of a media message or the effects of a drug on some disease, the criteria for evaluating its validity are the same. After reading about an experimental finding, you'd like to know to what extent you should consider the finding to be valid. Years ago, psychologists Richard Petty and Timothy Brock pointed out that *_____________________* must be evaluated in 4 different areas: 1. *______________________* - In any published experiment, there will usually be at least one comparison between experimental groups that is declared to be statistically significant. -The evaluation of the statistical validity of the experiment focuses on the validity of such a declaration. -Did the authors use the correct statistical test for their comparison? -Did they check to see that their data conformed to all of the assumptions behind those tests? -Did they make the correct conclusion about the significance of the result? -Usually, if an experiment is published in a reputable journal, the reader will be able to assume the statistical validity of the findings because experts in the field reviewed the article prior to publication. -Being able to make an independent judgment about statistical validity requires years of study in statistical techniques so it's fortunate that readers can usually assume safely that the statistical results for a published study are valid. 2. *____________________* - A crucial issue in evaluating any experiment has to do with whether the experimental treatment actually caused the effect that was observed. -The informed reader of an experiment will scrutinize the description of the experiment to determine if there were factors internal to the experiment itself—other than the treatment—that could have plausibly accounted for the results. -For example, if participants aren't randomly assigned to the experimental groups before the treatment is administered, the groups may be different in some way before the experiment even begins. -If that's the case, any difference at the end of the experiment might be due to those initial group differences instead of the treatment. -Another source of group differences other than the experimental treatment can creep into the procedures that are carried out while conducting a study. -For example, if a male experimenter administers the instructions for the experiment to one of the experimental groups while a female serves the same role for another group, this introduces another difference in the groups that could account for any experimental result. -It takes very careful planning for experimenters to guard against all of the potential threats to internal validity. 3. *_____________________* - After any experimental result is reported, the reader will want to know to what extent the results generalize to other people, places, times, or settings. -There's no way to know with certainty the answers to these questions without additional research. -Some critics of experimentation are fond of observing that results from laboratory studies are "artificial" and can't be generalized to any meaningful situation outside the lab. -While this sort of critique highlights the need to do more research to show the generalizability of the findings, it is hardly a devastating critique against experimentation in general. -Experiments are not designed primarily to show the generality of findings. -Instead, they are designed to show that under some set of specific conditions, one variable can cause another. -Once causality has been shown in one condition, more research can be conducted to explore the limitations or generality of the finding. 4. *_______________________* - After considering statistical, internal and external validity, readers need to think carefully about whether the variables used in the experiment actually reflect the variables that the researchers think they do. -That is to say, the experimental treatment may have caused an effect—but what was it about the treatment that caused the effect to emerge? -For example, in Chapter 5, you'll read about a study in which boys who watched media violence behaved less aggressively afterward than boys who watched non-violent programs. -The researcher who conducted the study concluded that viewing violence was cathartic or therapeutic, leading to a reduced need for boys to act aggressively after their viewing. -But even though the treatment was necessary to cause the effect, the results may have had absolutely nothing to do with whether the programs were violent or not. -As it turned out, the key variable may have actually been the extent to which boys liked the programs in each condition. -The fact that the boys who watched violence behaved less aggressively may have been due simply to the fact that they were watching programs they liked—not because the violence in those programs helped them to drain off their aggressive impulses. -The issue of what construct (liking vs. violence in this case) the variables in a study may represent is often one of the most interesting aspects of interpreting any experimental result.

*sample survey*

THE SAMPLE SURVEY: Primack's study on cannabis use employed 2 different methods—content analysis and a survey. Just as Coyne used a best sellers list to document substance use in books, Primack used content analysis to document lyrics that referred to cannabis in songs listed on the top charts in Billboard magazine. Content analysis in mass- communication research focuses on message content. But there are hosts of interesting research questions that can't be answered with this method. Many of the most important questions require researchers to investigate "people" instead of message content. One of the best ways of investigating people is to ask them questions directly. That's what Primack did when he asked 9th-graders for reports on their music exposure and their cannabis use in the last month. The method that is designed particularly for this purpose is the survey. It is often referred to as the *________________* because, in most cases, it is based on a random sample of some larger population of interest. The survey method is probably the most familiar to the average citizen. If you haven't actually participated in a survey as a respondent in the past month, you have almost certainly heard the results of at least 1 survey reported by the mass media. The quest by politicians and government officials for an accurate gauge of public opinion has served to refine the methods of survey research over the years. Today, sample surveys play a vital part in the governmental process. Mitt Romney and Barack Obama used sample surveys throughout their fierce battle for the presidency in 2012 to determine where they needed to concentrate their campaigning. The U.S. Census Bureau relies heavily on sample surveys to reveal information about the changing demographics in the population. In mass-communication research, surveys are invaluable in helping us to understand people's media habits. Through survey research, scholars now have a good sense of how much TV the average person watches, what types of content are most popular, and what people report about their own reactions to that content. The survey, as it turns out, is an excellent method for getting descriptive insight into a given phenomenon. Beyond simple descriptive data, the survey is an excellent way of exploring relationships between different variables. Suppose, for example, a researcher thought that children who tend to watch lots of television during a typical week also tend to be more obese. This relationship could be uncovered in a sample survey. Phone numbers in a given area might be selected randomly, and parents who answered the phone might be asked a number of questions about their child's weekly TV viewing. After getting estimates of the child's daily viewing, the researcher might also solicit estimates of each child's weight. After the researcher had collected these data, a statistical analysis could be applied to the data to see if the children who watch more TV during the week also tend to be the children who weigh more. Studies similar to the one described have actually been conducted. For example, a group of researchers writing in the Journal of the American Medical Association reported a survey of well over 50,000 women. They found that TV viewing was especially related to a significantly higher level of obesity and type 2 diabetes. These research findings are consistent with the results of the survey reported by Larry Tucker and Marilyn Bagwell. These authors surveyed nearly 5,000 adult females and found that obesity was twice as likely to occur among those who viewed 4 or more hours of TV per day as compared to those who viewed less than 1 hour of TV per day. More recently, a team of medical researchers examined the possible relationship between TV viewing and hypertension in obese children. Obese children who viewed over 4 hours of TV per day were over 3 times more likely to have hypertension than were the obese children who watched less than 2 hours of TV per day. Statistical analyses that reveal these types of findings are important tools of the mass-communication researcher. First, let's take a closer look at an example of an actual survey that was conducted to study media effects.

*causal relationships*

THE SEARCH FOR CAUSAL RELATIONSHIPS: Before considering the 3rd general method used by researchers in mass communication, it is important to understand that 1 of the principal goals of doing research on media effects is to explore the possibility that media messages bring about some change in people's thoughts, attitudes, emotions, behaviors, and so on. That is, the researcher is interested in documenting *_____________________* between media and people. Documenting causal relationships is not nearly as easy as one might suspect. 3 general criteria need to be met before a researcher can make the claim that 1 thing (the media message) is causing another (a change in attitude or behavior, for example).

*random sample*

The Sample: Every content analysis has to be done on a particular sample of messages. In this case, the researchers wanted to find out the content of popular adolescent novels. They decided to use one of the industry's most important lists for establishing the top novels at any given time: The New York Times Best Sellers list for children's books. They chose to examine the 40 most popular novels on the list that was published between June 22, 2008 and July 6, 2008. When the authors counted the number of pages that appeared in all of these novels, they had a total of 14,005 pages to analyze. When researchers rely on published lists such as the ones used in this study, there may be questions about how those lists were assembled. For example, in this case, the precise way in which this "best sellers" list is constructed is an inside secret that The New York Times refuses to reveal. Over the years, authors and publishers have criticized the list for various reasons and argued that it may not actually reflect the true "best sellers." Nevertheless, this list is highly influential in the publishing industry and the authors of this study believed the list to be useful for their purposes. Researchers must always think carefully about the way their sample of material is generated and what the conclusions drawn from the sample actually mean. In this study, the researchers had a manageable population of 40 novels and they chose to analyze every one. Sometimes, the population that a researcher wants to draw from is so large that a smaller random sample is chosen. For example, imagine that the researchers could be interested in generalizing across a 20-year period and had thousands of novels to consider. In that case, they may have decided to select a random sample from the total. A *________________* is one that permits every member of the population to have an equal chance of being selected in the sample. It is a crucial technique if the researcher wants to be able to generalize the results of the sample back to the larger population. In many content analysis problems, the population of interest is so large that it simply wouldn't be worth the time and expense to use all members. That would certainly be the case if the researchers of this study had several thousand novels to analyze.

*coder* *Manifest content* *Latent content*

What Is Content Analysis?: One researcher, Bernard Berelson, defined content analysis in the following way: "Content analysis is a research technique for the "objective", "systematic", and "quantitative" description of the "manifest" content of communication." Essentially, content analysis is a method that allows the researcher to describe messages in quantitative terms even though those messages are essentially verbal or non-quantitative in nature. Content analysis is "objective" in the sense that the method permits multiple researchers to examine the same content and come to identical conclusions. This is possible because the method is "systematic". That is, it specifies an unambiguous set of rules or procedures for coding the message content. Theoretically, any *_________* (a person who examines the content and classifies it into categories) who understands the rules or procedures will arrive at the same coding of the message content as any other coder. The data that result from a content analysis are "quantitative". That is, certain aspects of the content are coded and tallied in some quantitative way. This aspect of content analysis is important because it permits the researcher to conduct various statistical tests on the results of the coding. Finally, content analysis is concerned with the coding of manifest content rather than latent content. *__________________* refers to the material that actually appears, and requires a minimum of interpretation by the coder. *________________* is content that might become apparent after a coder has interpreted or "read between the lines" of the message before coding. For example, a coder might classify the following statement in a TV sitcom as a compliment: "Gee, Jerry, that's a real attractive outfit you're wearing." The surface meaning of the statement suggests that it should be coded as a compliment. However, it might be apparent from other cues in the program (perhaps the vocal tone of the person who spoke) that the statement was uttered with sarcasm and was not actually intended as a compliment. It might even take special knowledge about the character's personality in order to make a clear interpretation about whether the statement was sarcastic. If the coder classified the statement as an insult instead of as a compliment, that might be an example of coding the latent content. As you can probably begin to appreciate, the issue of manifest versus latent content is controversial among content analysts. Some researchers are inclined to code only the manifest content and discuss alternative interpretations after the data have been analyzed. Other researchers may be interested in coding the latent content directly, as long as agreement among different coders can be established. Although this overview may provide a general conceptual introduction to content analysis, chances are good that your understanding of this method is still a little vague. An example is in order. Suppose you believe that adolescent novels can potentially influence readers by setting certain expectations, norms, or standards for using alcohol or other illegal substances. In particular, suppose you think you have noticed that many popular novels directed to adolescents contain references to smoking, alcohol consumption, and drug use. You want to study the best-selling novels in some systematic fashion to determine whether your casual observation has any merit, but you aren't sure how to proceed. Content analysis is the method that can help you to learn whether your casual observation has any merit.

*content analysis*; *survey*; *experiment*

When I was about 14 years old, my grandmother came to visit for the summer. She loved to watch soap operas in the middle of the afternoon. My father didn't think much of soap opera content, and he didn't hesitate to voice his opinion that soap operas affect viewers in a negative way. The gist of his complaint was that he thought soap operas encouraged viewers to see their own lives as a constant melodrama. I was never sure that I agreed with my dad about the impact of soap operas on viewers, but his strong opinions on the subject made me wonder. *How would one ever go about discovering the effects of media content?* Having read Chapter 1, you may suspect that a good general answer to this question is to use a scientific approach. But what would such an approach entail? One of the main goals of this chapter is to introduce 3 specific methods that social scientists use to answer questions related to media impact. Taken together, these methods constitute the main arsenal of the media effects researcher. The 3 methods are: *____________________*, the *____________*, and the *___________________*.