Regression tests to expose change interaction errors

Marcel Böhme, Bruno C.D.S. Oliveira, Abhik Roychoudhury

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

40 Citations (Scopus)

Abstract

Changes often introduce program errors, and hence recent software testing literature has focused on generating tests which stress changes. In this paper, we argue that changes cannot be treated as isolated program artifacts which are stressed via testing. Instead, it is the complex dependency across multiple changes which introduce subtle errors. Furthermore, the complex dependence structures, that need to be exercised to expose such errors, ensure that they remain undiscovered even in well tested and deployed software. We motivate our work based on a well tested and stable project - GNU Coreutils - where we found that one third of the regressions take more than two (2) years to be fixed, and that two thirds of such long-standing regressions are introduced due to change interactions for the utilities we investigated. To combat change interaction errors, we first define a notion of change interaction where several program changes are found to affect the result of a program statement via program dependencies. Based on this notion, we propose a change sequence graph (CSG) to summarize the control-flow and dependencies across changes. The CSG is then used as a guide during program path exploration via symbolic execution - thereby efficiently producing test cases which witness change interaction errors. Our experimental infrastructure was deployed on various utilities of GNU Coreutils, which have been distributed with Linux for almost twenty years. Apart from finding five (5) previously unknown errors in the utilities, we found that only one in five generated test cases exercises a sequence that is critical to exposing a change-interaction error, while being an order of magnitude more likely to expose an error. On the other hand, stressing changes in isolation only exposed half of the change interaction errors. These results demonstrate the importance and difficulty of change dependence-aware regression testing.

Original languageEnglish
Title of host publication2013 9th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE 2013 - Proceedings
Pages334-344
Number of pages11
DOIs
Publication statusPublished - 2013
Externally publishedYes
EventJoint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2013 - Saint Petersburg, Russian Federation
Duration: 18 Aug 201326 Aug 2013
Conference number: 9th
http://esec-fse.inf.ethz.ch/

Conference

ConferenceJoint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2013
Abbreviated titleESEC/FSE 2013
Country/TerritoryRussian Federation
CitySaint Petersburg
Period18/08/1326/08/13
Internet address

Keywords

  • Graph-based search
  • Test generation

Cite this