"Automated debugging considered harmful" considered harmful: a user study revisiting the usefulness of spectra-based fault localization techniques with professionals using real bugs from large systems

Xin Xia, Lingfeng Bao, David Lo, Shanping Li

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

31 Citations (Scopus)

Abstract

Due to the complexity of software systems, bugs are inevitable. Software debugging is tedious and time consuming. To help developers perform this crucial task, a number of spectra-based fault localization techniques have been proposed. In general, spectra-based fault localization helps developers to find the location of a bug given its symptoms (e.g., program failures). A previous study by Parnin and Orso however implies that several assumptions made by existing work on spectra-based fault localization do not hold in practice, which hinders the practical usage of these tools. Moreover, a recent study by Xie et al. claims that spectra-based fault localization can potentially "weaken programmers' abilities in fault detection". Unfortunately, these studies are performed either using only 2 bugs from small systems (Parnin and Orso's study) or synthetic bugs injected into toy programs (Xie et al.'s study), only involve students, and use dated spectra-based fault localization tools. Thus, the question whether spectra-based fault localization techniques can help professionals to improve their debugging efficiency in a reasonably large project is still insufficiently answered. In this paper, we perform a more realistic investigation of how professionals can use and benefit from spectra-based fault localization techniques. We perform a user study of spectra-based fault localization with a total of 16 real bugs from 4 reasonably large open-source projects, with 36 professionals, amounting to 80 recorded debugging hours. The 36 professionals are divided into 3 groups, i.e., those that use an accurate fault localization tool, use a mediocre fault localization tool, and do not use any fault localization tool. Our study finds that both the accurate and mediocre spectra-based fault localization tools can help professionals to save their debugging time, and the improvements are statistically significant and substantial. We also discuss implications of our findings to future directions of spectra-based fault localization.

Original languageEnglish
Title of host publicationProceedings - 2016 IEEE International Conference on Software Maintenance and Evolution, ICSME 2016
Subtitle of host publicationRaleigh, North Carolina, USA 2-10 October 2016
EditorsBram Adams, Denys Poshyvanyk
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages267-277
Number of pages11
ISBN (Electronic)9781509038060, 9781509038077
DOIs
Publication statusPublished - 2016
Externally publishedYes
EventIEEE International Conference on Software Maintenance and Evolution 2016 - Raleigh, United States of America
Duration: 2 Oct 201610 Oct 2016
Conference number: 32nd
http://icsme2016.github.io/

Conference

ConferenceIEEE International Conference on Software Maintenance and Evolution 2016
Abbreviated titleICSME 2016
CountryUnited States of America
CityRaleigh
Period2/10/1610/10/16
Internet address

Keywords

  • Automated debugging
  • Empirical study
  • Spectra-based fault localization
  • User study

Cite this

Xia, X., Bao, L., Lo, D., & Li, S. (2016). "Automated debugging considered harmful" considered harmful: a user study revisiting the usefulness of spectra-based fault localization techniques with professionals using real bugs from large systems. In B. Adams, & D. Poshyvanyk (Eds.), Proceedings - 2016 IEEE International Conference on Software Maintenance and Evolution, ICSME 2016: Raleigh, North Carolina, USA 2-10 October 2016 (pp. 267-277). [7816473] IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICSME.2016.67