Chapter 4 - General Issues in Research Design

Groups

• A variety of groups may also be the units of analysis for criminal justice research (not the same as studying the individuals within a group) • E.g., study all juvenile gangs in a city to learn the differences between big gangs and small ones, between gangs selling drugs and gangs stealing cars, and so forth • E.g., households, city blocks, census tracts, cities, counties, and other geographic regions

Internal Validity

• An observed association between two variables has internal validity if the relationship is, in fact, causal and not due to the effects of one or more other variables • e.g., drug users sentenced to probation rearrested less often than those sentenced to prison • conclusion: prison causes recidivism • Internal validity problems result from nonrandom or systematic error (threats usually arise from the effects of one or more other variables) • Relates to the third requirement for establishing a causal relationship: eliminating other possible explanations for the observed relationship

Individuals

• Any variety of individuals may be the units of analysis in criminal justice research • The norm of generalized understanding in social science should suggest that scientific findings are most valuable when they apply to all kinds of people • In practice, researchers seldom study all kinds of people • As the units of analysis, individuals may be considered in the context of their membership in different groups • E.g., police, victims, defendants in criminal court, correctional inmates, gang members, and active burglars

Types of Empirical Questions: (2) Causal Qs - Why?

• Asking how does a change in one variable have an impact on another variable (a much stronger claim than description)

Types of Empirical Questions: (1) descriptive Q's - What?

• Attempt to describe a phenomenon (situation, etc)

The Time Dimension

• Because time order is a requirement for causal inferences, the time dimension of research requires careful planning - time sequence is critical in determining causation • Time is involved in the generalizability of research findings • In general, observations may be made more or less at one time point, or they may be deliberately stretched over a longer period • Observations made at more than one time point can look forward or backward

Causation in the Social Sciences

• Causation is the focus of explanatory research • Typically, we do that by specifying the causes for the way things are: some things are caused by other things • Cause in social science is inherently probabilistic • Probabilistic: reflecting the causal reasoning that certain factors make outcomes more or less likely to happen (having been arrested as a juvenile makes it more likely that one will be arrested as an adult)

Introduction

• Causation, units, and time are key elements in planning a research study • In practice, all aspects of research design are interrelated

Construct Validity

• Construct validity refers to generalizing from what we observe and measure to the real-world things in which we are interested (concerned with how well an observed relationship between variables that a researcher has measured represents the underlying causal process of interest) - e.g., supervision • Closely related to the issues in measurement • Construct validity is a frequent problem in applied studies, in which researchers may oversimplify complex policies and policy goals

Criteria for Causality

• Criteria for assessing an idiographic explanation (according to Joseph Maxwell): 1) How credible and believable it is (relates to logic as one of the foundations of science - demand that our explanations make sense) 2) Whether alternative explanations ("rival hypotheses") were seriously considered and found wanting (when all other possibilities have been eliminated, the remaining explanation, however improbably, must be the truth) • 3 specific criteria for causality, regarding nomothetic explanation (by William Shadish, Thomas Cook, and Donald Campbell) 1) Two variables must vary together - they must be empirically correlated (how strong the empirical relationship must be for that relationship to be considered causal?) 2) The cause must occur before the effect (often, the time order connecting two variables is simply unclear/ even when the time order seems clear, exceptions may be found) 3) The empirical correlation between cause and effect is not due to some other factor

The Time Dimension Summarized

• Cross-sectional study as a snapshot • Trend study as a slide show • Panel study as a motion picture

Introducing Scientific Realism

• Doing research to find what causes what most often reflects nomothetic concerns (to find causal explanations that apply generally to situations beyond those we actually study in our research) • Researchers and public officials are also often interested in understanding specific causal mechanisms in more narrowly defined situations - idiographic mode of explanation • Scientific realm bridges idiographic and nomothetic approaches to explanation by seeking to understand how causal mechanisms operate in specific context • The scientific realist approach views these other possible influences as contexts in which causal mechanisms operate • A scientific realism approach would recognize that drug use and crime co-occur in some contexts but not in others • Scientific realism focuses our attention on very specific questions, it seems idiographic • But analysis and interpretation addressed a more general question of causation

Issues of logic

• Ecological fallacy • Individual fallacy - using focused anecdotal evidence to support a general statement (e.g., skytrain attacks) • Reductionism

The Ecological Fallacy

• Ecological fallacy refers to the danger of making assertions about individuals as the unit of analysis based on the examination of groups or other aggregations (such assertions are often made in connection with causation, where researchers observe associations between aggregate units and make statements about causality between individual units) • e.g., poor areas have more crime, therefore poor people commit more crime

Time Dimension: Cross-Sectional Studies

• Exploratory and descriptive studies are often cross-sectional studies • Cross-sectional study: data collected/Look at a phenomenon at a single time point • Cross-sectional studies for explanatory or evaluation purposes have an inherent problem (inferring cause requires that a cause precede an effect in time, but cross-sectional studies produce observations made at only one time) - problem in establishing temporal sequence • e.g., a single wave of a criminal victimization survey is a descriptive cross-sectional study that estimates how many people have been victims of crime in a given time

Organizations

• Formal political or social organizations may also be the units of analysis in criminal justice research • E.g., correctional facilities, police departments, courtrooms, probation offices, drug treatment facilities, and victim services agencies • When social groups or formal organizations are the units of analysis, their characteristics are often derived from the characteristics of their individual members • some studies involve descriptions or explanations of more than one unit of analysis

External Validity

• In a general sense, external validity is concerned with whether research findings from one study can be reproduced (replicated) in another study, often under different conditions • do the findings apply equally in different settings (locales, cities, populations)?

Approximating Longitudinal Studies

• It may be possible to draw conclusions about processes that take place over time even when only cross-sectional data are available • Sometimes, cross-sectional data imply processes that occur over time on the basis of simple logic

Time Dimension: Longitudinal Studies

• Longitudinal studies: designed to permit observations over an extended period • 3 special types of longitudinal studies: trend, cohort, and panel studies • Trend studies look at changes within some general population over time (e.g., UCR) • Cohort studies examine more specific populations (cohorts) as they change over time (typically, a cohort is an age group but it can also be based on some other time grouping) o Cohorts are often defined as a group of people who enter or leave an institution at the same time, such as persons entering a drug treatment center during July • Panel studies are similar to trend and cohort studies except that observations are made on the same set of people on two or more occasions (panel studies are often used in evaluation research, in which the same persons are interviewed both before and after a new program is introduced) • Longitudinal studies tend to be expensive and difficult to conduct, and panel studies face a special problem: panel attrition (some of the respondents studied in the first wave of study may not participate in later waves - the danger is that those who drop out of the study may differ in some way compared to those who remain in the study, and may thereby distort the results of the study)

Reductionism

• Reductionism is an overly strict limitation on the kinds of concepts and variables to be considered as causes in explaining the broad range of human behaviour represented by crime and criminal justice policy • Reductionism of any type tends to suggest that particular units of analysis or variables are more relevant than others • Reductionism involves the use of inappropriate units of analysis (the appropriate unit of analysis for a given research question is not always clear and is often debated by social scientists, especially across disciplinary boundaries) • reductionism - inappropriate focus. Failing to see the myriad of possible factors causing the situation being studied o academics tend to interpret within their discipline o inappropriate stress on a narrow range of variables

Time Dimension: Retrospective and Prospective Studies

• Retrospective research, which asks people to recall their pasts, is another common way of approximating observations over time • The danger in this technique (sometimes, people have faulty memories, or lie + records of the past may be unavailable, incomplete, or inaccurate) • A more fundamental problem in retrospective research hinges on how subjects are selected and how subject selection affects the kinds of questions such studies can address • Prospective approach - longitudinal study that follows subjects forward in time (hard to do) • "Looking back over the careers of adult criminals exaggerates the prevalence of stability. Looking forward from youth reveals the success and failures, including adolescent delinquents who go on to be normal functioning adults"

Validity ad Causal Inference

• Scientists assess the truth of statements about cause by considering threats to validity • Validity: whether statements about cause or measures are correct (valid) or false (invalid) • Emphasize that approximate is an important word because one can never be absolutely certain about cause • It is almost never possible to conclusively establish the validity of causal inference • Validity threats: possible sources of false conclusions about cause or measurement • Four general categories of validity (statistical conclusion validity, internal validity, construct validity, and external validity)

Social Artifacts

• Social artifacts = the products of social beings and their behaviour • E.g., stories about crime in newspapers and magazines or on television, social interactions • Records of different types of social interactions are common units of analysis in criminal justice research (e.g., criminal history records, meetings of community anticrime groups, presentence investigations, and interactions between police and citizens - each example requires information about individuals but social interactions between people are the units of analysis)

Statistical Conclusion Validity

• Statistical conclusion validity: our ability to determine whether a change in the suspected cause (IV) is statistically associated with a change in the suspected effect (DV) (are two variables related to each other?) • Basing conclusions on a small number of cases is a common threat to statistical conclusion validity (researchers cannot have much confidence in statement about cause if their findings are based on a small number of cases) -> *sample size is important • Threats to statistical conclusion validity might also have the opposite effect of suggesting that covariation is present when, in fact, there is no cause-and-effect relationship (the reasons for this are again somewhat technical and require a basic understanding of statistical inference - e.g., superstitious behaviour exhibited by gamblers)

Does Drug Use Cause Crime?

• Temporal order: which one comes first, drug use or crime? (no conclusive answer) • Statistical association between drug use and crime clearly exists • Drug use and crime (as well as delinquency) are each deviant activities produced by other underlying causes o A statistical association between drug use and crime clearly exists (but the presence of other factors indicates that the relationship is not directly causal, thus bringing into question the internal validity of causal statements about drug use and crime) • Because there is no simple way to describe either construct, searching for a single cause-and-effect relationship misrepresents a complex causal process • Both construct and external validity are concerned with generalizations • The issue of external validity comes into sharper focus when we shift from basic research that seeks to uncover fundamental causal relationships to criminal justice policy • Basic and applied research on the relationships among drug use and crime readily illustrates threats to the validity of causal inference (often difficult to find a relationship because there is so much variatioin in drug use and crime participation (statistical conclusion validity threat)) • A large number of studies have demonstrated that, when statistical relationships are found, both drug use and crime can be attributed to other, often multiple, causes (internal validity threat) • Different patterns among different population groups mean that there are no readily identifiable cause-and-effect constructs (construct validity) • Because of these differences, policies developed to counter drug use among the population as a whole cannot be expected to have much of an impact on serious crime (external validity) • No simple causal connection

Validity and Causal Inference Summarized

• The four types of validity threats can be grouped into two categories: bias and generalizability • Internal and statistical conclusion validity threats are related to systematic and nonsystematic bias (problems with statistical procedures produce nonsystematic bias, while an alternative explanation for an observed relationship is an example of systematic bias) • In either case, bias calls into question the inference that some cause produced some effect • Failing to consider the more general cause-and-effect constructs that operate in an observed cause-and-effect relationship results in research findings that cannot be generalized to real-world behaviours and conditions • A cause-and-effect relationship observed in one setting or at one time may not operate in the same way in a different setting or at a different time 1. How large and reliable is the covariation between the presumed cause and effect? 2. Is the covariation causal, or would the same covariation have been obtained without the treatment 3. How generalizable is the locally embedded causal relationship over varied persons, treatments, observations, and settings? 4. What general constructs are involved in the persons, settings, treatments, and observations used in the experiment?

Units of Analysis

• To avoid mistaken inferences, researchers must carefully specify the people or phenomena that will be studied • Individual people are often units of analysis (e.g., police, victims, defendants, inmates etc) • Groups as units of analysis - multiple persons with same characteristics (e.g., gangs, cities, counties, etc) • Organizations - formal groups with established leaders and rules (e.g., prisons, police departments, courtrooms, drug treatment facilities, etc) • Social artifacts - products of social beings and their behaviour (e.g., stories in newspapers, posts on the internet, photographs on crime scenes, incident reports, police/citizen interactions) • Units of analysis: the tings - what or whom - being studied in a research project • Units of analysis in a study are typically also the units of observation

Necessary and Sufficient Causes

• Within the probabilistic model, it is useful to distinguish two types of causes: necessary and sufficient causes • Necessary cause: a condition that must be present for the effect to follow/occur (e.g., being charged is a necessary cause to be convicted) • Sufficient cause: a condition that more or less guarantees the effect in question (e.g., pleading guilty is sufficient cause to being convicted) • We seldom discover causes that are both necessary and sufficient • Most causal relationships that criminal justice researchers work with are probabilistic and partial - we are able to partially explain cause and effect in some percentage of cases we observe

Criterion for causality: no plausible alternative explanations

• correlation between cause and effect is not due to some other factor (a third variable) • problem: spurious relationships • No experimental studies to eliminate rival plausible explanations

Correlational vs. causal relationships

• correlational: two variables varying in a synchronized manner • causal: one variable responsible for change in another variable

validity

• in science and statistics, validity is the extent to which a concept, statement, or measurement is well-founded and corresponds accurately to the real world. The word "valid" is derived from the Latin validus, meaning strong

Criterion for causality: temporal precedence

• the presumed cause (IV) must occur before the presumed effect (DV) • problem: time order is often unclear (e.g., drug use and crime)

Criterion for causality: empirical relationship, i.e., the variables are correlated

• the variables covary (co-relate) • e.g., more smoking is related to more lung cancer • inverse relationships also possible • e.g., smoking is "relaxing" (reduces jitters due to nicotine withdrawal symptoms) • problem: • There aren't many "perfect" correlational relationships, ie. the things of interest always occur together

Validity - bottom line

• these 4 validities are the main criteria for judging the informativeness of a study (that is, does the study as represented in statements based on it, provide us with valid information) • relative importance of each type of validity depends on study purpose

validity of statements

• whether statements about a given cause and effect are true (valid) or false (invalid) • different from validity of measures • 4 subtypes