Why experiment?

Randomization and evidence-based practice: Why experiment?

From 1950 to 2005, the age-adjusted death rate in the U.S. declined by nearly 45 percent (Figure 1). Much of this success was the result of a combination of promising ideas and rigorous evaluation. Consider, for example, the fight against polio. Starting with the discovery in 1908 that polio was caused by a virus, researchers tried numerous different ways of vaccinating children against the disease. Yet evaluations showed these early vaccines were not effective, which led to more bench science, which eventually led to Dr. Jonas Salk's discovery in 1952. Even then, the Salk vaccine only went into widespread use after it was confirmed in 1955 through a series of clinical trials to be safe and effective - that is, a promising idea that actually works in practice.

Figure 1: Deaths from Homicide and all causes, 1950-2005 (Source: National Center for Health Statistics)

We have not been as successful in figuring out how to protect people from crime and violence. Despite some cyclical variation over time, the nation's murder rate in 2005 was roughly 20 percent higher than it was in 1950. The problem is not a lack of ideas or new programs. Rather, the problem is that in social policy, unlike the ideal in medicine and public health, new programs are rarely implemented in ways that can be rigorously evaluated.

We need rigorous evaluation because our theories about what is likely to work are far from perfect. When developers, advocates, and elected officials pressed for a system of government housing in the 1930s, few imagined the result would be projects like the high-rises on Chicago's South and West sides. While the problems created by high-rise housing projects quickly became obvious, the success or failure of most programs is more difficult to determine. From its inception, the Head Start program was criticized as having little impact on children, but the program's benefits were later discovered to have been obscured by the extra home and neighborhood disadvantages experienced by participating children.

Overconfidence is also a risk. In the criminal justice area, claims to have discovered the Salk vaccine for crime and violence are not uncommon, particularly for programs or policies that were adopted during the 1990s - a period when crime happened to be dropping everywhere (even in places that did not adopt any new criminal justice or social policy innovations) for reasons that even today remain only partially understood.

The lesson is that progress in addressing the problem of crime and violence is extremely difficult without learning from experience - we require guidance about what programs work, for whom, why, and how they can be improved. The vision of the University of Chicago Crime Lab is to partner with local governments and community organizations to conduct and rigorously evaluate-first in Chicago, and eventually nationwide - promising pilot programs to reduce the toll that crime and violence impose on American society every year. This accumulated set of evaluation evidence will help places learn from one another about "best practices" for reducing the social costs of crime and violence.