The impact of class rebalancing techniques on the performance and interpretation of defect prediction models

Chakkrit Tantithamthavorn, Ahmed E. Hassan, Kenichi Matsumoto

Research output: Contribution to journalArticleResearchpeer-review

16 Citations (Scopus)

Abstract

Defect models that are trained on class imbalanced datasets (i.e., the proportion of defective and clean modules is not equally represented) are highly susceptible to produce inaccurate prediction models. Prior research compares the impact of class rebalancing techniques on the performance of defect models but arrives at contradictory conclusions due to the use of different choice of datasets, classification techniques, and performance measures. Such contradictory conclusions make it hard to derive practical guidelines for whether class rebalancing techniques should be applied in the context of defect models. In this paper, we investigate the impact of class rebalancing techniques on performance measures and the interpretation of defect models. We also investigate the experimental settings in which class rebalancing techniques are beneficial for defect models. Through a case study of 101 datasets that span across proprietary and open-source systems, we conclude that the impact of class rebalancing techniques on the performance of defect prediction models depends on the used performance measure and the used classification techniques. We observe that the optimized SMOTE technique and the under-sampling technique are beneficial when quality assurance teams wish to increase AUC and Recall, respectively, but they should be avoided when deriving knowledge and understandings from defect models.

Original languageEnglish
Number of pages22
JournalIEEE Transactions on Software Engineering
DOIs
Publication statusAccepted/In press - 2019

Keywords

  • Analytical models
  • class rebalancing techniques
  • Context modeling
  • empirical investigation
  • experimental design
  • Guidelines
  • Open source software
  • Predictive models
  • software defect prediction
  • Software quality assurance
  • Training

Cite this