Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores

Thomas A. Loughran, Theodore Wilson, Daniel S. Nagin, Alex R. Piquero

Research output: Contribution to journalArticleResearchpeer-review

29 Citations (Scopus)

Abstract

Objectives: Propensity score methods rely on an untestable assumption of unconfoundedness for making causal inference. Yet, empirical applications using propensity scores in criminology routinely invoke this assumption without careful scrutiny. Methods: We use a dataset with a wide range of observable, potential confounders, which allows us to evaluate recidivism outcomes for adolescent offenders who are sentenced to either placement or probation. We then systematically withhold important known confounders from the matching process to demonstrate the effectiveness of sensitivity checks in sizing up the robustness of these treatment effect estimates in the case where hidden biases clearly exist. Results: We find important variability in the estimated treatment effect, and a large degree of imbalance in ‘unobserved’ covariates, which we did not explicitly control for. The hidden biases observed in our controlled analysis would have at least been suggested in an actual application by the low gamma statistics that attended our analysis, a statistic that is not reported in most criminological applications of propensity score analysis. Conclusions: Researchers who use propensity score methods should openly discuss potential limitations of their analysis due to hidden bias and report bias sensitivity checks based on the gamma statistic when statistically significant treatment effects are reported.

Original languageEnglish
Pages (from-to)631-652
Number of pages22
JournalJournal of Experimental Criminology
Volume11
Issue number4
DOIs
Publication statusPublished - 1 Dec 2015
Externally publishedYes

Keywords

  • Hidden biases
  • Incarceration
  • Propensity score matching
  • Recidivism

Cite this