Supervised vs unsupervised models: a holistic look at effort-aware just-in-time defect prediction

Qiao Huang, Xin Xia, David Lo

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

38 Citations (Scopus)


Effort-aware just-in-time (JIT) defect prediction aims at finding more defective software changes with limited code inspection cost. Traditionally, supervised models have been used; however, they require sufficient labelled training data, which is difficult to obtain, especially for new projects. Recently, Yang et al. proposed an unsupervised model (LT) and applied it to projects with rich historical bug data. Interestingly, they reported that, under the same inspection cost (i.e., 20 percent of the total lines of code modified by all changes), it could find more defective changes than a state-of-the-art supervised model (i.e., EALR). This is surprising as supervised models that benefit from historical data are expected to perform better than unsupervised ones. Their finding suggests that previous studies on defect prediction had made a simple problem too complex. Considering the potential high impact of Yang et al.'s work, in this paper, we perform a replication study and present the following new findings: (1) Under the same inspection budget, LT requires developers to inspect a large number of changes necessitating many more context switches. (2) Although LT finds more defective changes, many highly ranked changes are false alarms. These initial false alarms may negatively impact practitioners' patience and confidence. (3) LT does not outperform EALR when the harmonic mean of Recall and Precision (i.e., F1-score) is considered. Aside from highlighting the above findings, we propose a simple but improved supervised model called CBS. When compared with EALR, CBS detects about 15% more defective changes and also significantly improves Precision and F1-score. When compared with LT, CBS achieves similar results in terms of Recall, but it significantly reduces context switches and false alarms before first success. Finally, we also discuss the implications of our findings for practitioners and researchers.

Original languageEnglish
Title of host publicationProceedings - 2017 IEEE International Conference on Software Maintenance and Evolution, ICSME 2017
Subtitle of host publication19–22 September 2017 Shanghai, China
EditorsLu Zhang, Thomas Zimmermann
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages12
ISBN (Electronic)9781538609927, 9781538609934
Publication statusPublished - 2017
Externally publishedYes
EventIEEE International Conference on Software Maintenance and Evolution 2017 - Shanghai, China
Duration: 17 Sep 201724 Sep 2017
Conference number: 33rd (Proceedings)


ConferenceIEEE International Conference on Software Maintenance and Evolution 2017
Abbreviated titleICSME 2017
Internet address


  • Bias
  • Change classification
  • Cost effectiveness
  • Evaluation

Cite this