Evaluating defect prediction approaches using a massive set of metrics: an empirical study

Xiao Xuan, David Lo, Xin Xia, Yuan Tian

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

24 Citations (Scopus)


To evaluate the performance of a within-project defect prediction approach, people normally use precision, recall, and F-measure scores. However, in machine learning literature, there are a large number of evaluation metrics to evaluate the performance of an algorithm, (e.g., Matthews Correlation Coefficient, G-means, etc.), and these metrics evaluate an approach from different aspects. In this paper, we investigate the performance of within-project defect prediction approaches on a large number of evaluation metrics. We choose 6 state-of-the-art approaches including naive Bayes, decision tree, logistic regression, kNN, random forest and Bayesian network which are widely used in defect prediction literature. And we evaluate these 6 approaches on 14 evaluation metrics (e.g., G-mean, F-measure, balance, MCC, J-coefficient, and AUC). Our goal is to explore a practical and sophisticated way for evaluating the prediction approaches comprehensively. We evaluate the performance of defect prediction approaches on 10 defect datasets from PROMISE repository. The results show that Bayesian network achieves a noteworthy performance. It achieves the best recall, FN-R, G-mean1 and balance on 9 out of the 10 datasets, and F-measure and J-coefficient on 7 out of the 10 datasets.

Original languageEnglish
Title of host publicationThe 30th Annual ACM Symposium on Applied Computing
Subtitle of host publicationSalamanca, Spain April 13-17, 2015
EditorsAlessio Bechini, Jiman Hong
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages4
ISBN (Electronic)9781450331968
Publication statusPublished - 2015
Externally publishedYes
EventACM Symposium on Applied Computing 2015 - Salamanca, Spain
Duration: 13 Apr 201517 Apr 2015
Conference number: 30th
https://dl.acm.org/doi/proceedings/10.1145/2695664 (Proceedings)


ConferenceACM Symposium on Applied Computing 2015
Abbreviated titleSAC 2015
Internet address


  • Defect prediction
  • Evaluation metric
  • Machine learning

Cite this