CoREBench: studying complexity of regression errors

Marcel Böhme, Abhik Roychoudhury

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

41 Citations (Scopus)


Intuitively we know, some software errors are more complex than others. If the error can be fixed by changing one faulty statement, it is a simple error. The more substantial the fix must be, the more complex we consider the error. In this work, we formally define and quantify the complexity of an error w.r.t. the complexity of the error's least complex, correct fix. As a concrete measure of complexity for such fixes, we introduce Cyclomatic Change Complexity which is inspired by existing program complexity metrics. Moreover, we introduce CoREBench, a collection of 70 regression errors systematically extracted from several opensource C-projects and compare their complexity with that of the seeded errors in the two most popular error benchmarks, SIR and the Siemens Suite. We find that seeded errors are significantly less complex, i.e., require significantly less substantial fixes, compared to actual regression errors. For example, among the seeded errors more than 42% are simple compared to 8% among the actual ones. This is a concern for the external validity of studies based on seeded errors and we propose CoREBench for the controlled study of regression testing, debugging, and repair techniques.

Original languageEnglish
Title of host publication2014 International Symposium on Software Testing and Analysis (ISSTA 2014) - Proceedings
EditorsCorina S. Pasareanu, Darko Marinov
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages11
ISBN (Electronic)9781450326452
Publication statusPublished - 2014
Externally publishedYes
EventInternational Symposium on Software Testing and Analysis 2014 - San Jose, United States of America
Duration: 21 Jul 201425 Jul 2014
Conference number: 23rd


ConferenceInternational Symposium on Software Testing and Analysis 2014
Abbreviated titleISSTA 2014
CountryUnited States of America
CitySan Jose
Internet address


  • Coupling effect
  • Error complexity
  • Regression

Cite this

Böhme, M., & Roychoudhury, A. (2014). CoREBench: studying complexity of regression errors. In C. S. Pasareanu, & D. Marinov (Eds.), 2014 International Symposium on Software Testing and Analysis (ISSTA 2014) - Proceedings (pp. 105-115). Association for Computing Machinery (ACM).