A Formal Explainer for Just-In-Time Defect Predictions

Just-in-Tim e (JIT) defect prediction has been proposed to help teams prioritize the limited resources on the most risky commits (or pull requests), yet it remains largely a black box, whose predictions are not explainable or actionable to practitioners. Thus, prior studies have applied various model-agnostic techniques to explain the predictions of JIT models. Yet, explanations generated from existing model-agnostic techniques are still not formally sound, robust, and actionable. In this article, we propose FoX, a Formal eXplainer for JIT Defect Prediction, which builds on formal reasoning about the behavior of JIT defect prediction models and hence is able to provide provably correct explanations, which are additionally guaranteed to be minimal. Our experimental results show that FoX is able to efficiently generate provably correct, robust, and actionable explanations, while existing model-agnostic techniques cannot. Our survey study with 54 software practitioners provides valuable insights into the usefulness and trustworthiness of our FoX approach; 86% of participants agreed that our approach is useful, while 74% of participants found it trustworthy. Thus, this article serves as an important stepping stone towards trustable explanations for JIT models to help domain experts and practitioners better understand why a commit is predicted as defective and what to do to mitigate the risk.