A new paper by University of Virginia law Prof. Megan Stevenson surveys more than 50 years of "randomized controlled trials" (RCTs) in criminal justice research and argues that almost no interventions have lasting benefit, and the ones that do don't replicate in other settings.
RCT, a form of experiment used to control factors not under the direct control of researchers, is often called the gold standard o research methods.
Writing in the Boston University Law Review, Stevenson says that the relatively few RCT studies of anticrime efforts that survive the academic review process "are biased toward showing that the intervention evaluated was more successful than it actually was."
One example cited by Stevenson is the widely publicized Project Hawaii Opportunity Probation with Enforcement (“Project HOPE”). It features a "swift, certain, and fair” punishment model in which a probation violation such as a positive drug test may result in a certain and immediate, but relatively mild sanction, such as twenty-four hours in jail.
One randomized controlled trial study showed that Project HOPE led to large reductions in both drug use and time incarcerated, with long-lasting effects.
The National Institute of Justice funded new studies to replicate Project HOPE’s success across five sites. Stevenson says. "The results were not promising: “swift, certain, and fair” sanctioning did not offer any detectable improvements over the status quo. While jurisdictions may continue to operate in a HOPE-like fashion, the balloon of optimism has largely deflated."
Stevenson also discusses summer jobs programs for teens, saying that their "reductions in crime and criminal justice involvement are meaningful benefits. But note that there is no evidence that this intervention leads to wholesale change in youth trajectories. Summer jobs do not appear to increase average wages or employment after completion of the program, nor do they increase educational outcomes."
Stevenson's paper is described as "flawed" by Thomas Abt of the Center for the Study and Practice of Violence Prevention at the University of Maryland.
In a series of tweets on X, Abt says that, "What Stevenson misses is that individual incremental changes may look puny in isolation, but when aggregated they add up to real progress."
Abt complains that Stevenson, "rather than delv(ing) seriously into the world of systematic reviews and meta-analyses, (she) relies on a single study from 2006, then buttresses it with anecdotal examples. She superficially dismisses large bodies of RCT evidence indicating that strategies like hot spots policing and cognitive behavioral therapy produce modest to moderate anti-crime benefits."
In a response, Stevenson says, "I did not ignore the literature on hotspots, I just characterized the effect size as 'small,' as did the authors of the metanalyses I cited — who, by the way, are world-class experts on policing and major proponents of hotspots."
She adds, "If big claim to fame of RCTs is that it shows that sending police to certain street corners slightly lowers crime in those street corners — saying little about spillover effects or long-term/general equilibrium effects — I think many would agree this is underwhelming."