BOAT: an experimental platform for researchers to comparatively and reproducibly evaluate bug localization techniques

Xinyu Wang, David Lo, Xin Xia, Xingen Wang, Pavneet Singh Kochhar, Yuan Tian, Xiaohu Yang, Shanping Li, Jianling Sun, Bo Zhou

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

5 Citations (Scopus)


Bug localization refers to the process of identifying source code files that contain defects from descriptions of these de-fects which are typically contained in bug reports. There have been many bug localization techniques proposed in the literature. However, often it is hard to compare these tech-niques since different evaluation datasets are used. At times the datasets are not made publicly available and thus it is dificult to reproduce reported results. Furthermore, some techniques are only evaluated on small datasets and thus it is not clear whether the results are generalizable. Thus, there is a need for a platform that allows various techniques to be compared with one another on a common pool containing a large number of bug reports with known defective source code files. In this paper, we address this need by propos-ing our Bug lOcalization experimental plATform (BOAT). BOAT is an extensible web application that contains thou-sands of bug reports with known defective source code files. Researchers can create accounts in BOAT, upload executa-bles of their bug localization techniques, and see how these techniques perform in comparison with techniques uploaded by other researchers, with respect to some standard eval-uation measures. BOAT is already preloaded with several bug localization techniques and thus researchers can direct-ly compare their newly proposed techniques against these existing techniques.

Original languageEnglish
Title of host publication36th International Conference on Software Engineering (ICSE Companion 2014) - Proceedings
Subtitle of host publicationMay 31 – June 7, 2014 Hyderabad, India
EditorsPankaj Jalote, Lionel Briand, André van der Hoek
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages4
ISBN (Print)9781450327688
Publication statusPublished - 2014
Externally publishedYes
EventInternational Conference on Software Engineering 2014 - Hyderabad, India
Duration: 31 May 20147 Jun 2014
Conference number: 36th


ConferenceInternational Conference on Software Engineering 2014
Abbreviated titleICSE 2014
Internet address


  • Benchmark
  • BOAT
  • Bug localization
  • Experimental platform

Cite this