How to build repeatable experiments

Gregory Gay, Tim Menzies, Bojan Cukic, Burak Turhan

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

2 Citations (Scopus)

Abstract

The mantra of the PROMISE series is "repeatable, improvable, maybe refutable" software engineering experiments. This community has successfully created a library of reusable software engineering data sets. The next challenge in the PROMISE community will be to not only share data, but to share experiments. Our experience with existing data mining environments is that these tools are not suitable for publishing or sharing repeatable experiments. OURMINE is an environment for the development of data mining experiments. OURMINE offers a succinct notation for describing experiments. Adding new tools to OURMINE, in a variety of languages, is a rapid and simple process. This makes it a useful research tool. Complicated graphical interfaces have been eschewed for simple command-line prompts. This simplifies the learning curve for data mining novices. The simplicity also encourages large scale modification and experimentation with the code. In this paper, we show the OURMINE code required to reproduce a recent experiment checking how defect predictors learned from one site apply to another. This is an important result for the PROMISE community since it shows that our shared repository is not just a useful academic resource. Rather, it is a valuable resource industry: companies that lack the local data required to build those predictors can use PROMISE data to build defect predictors.

Original languageEnglish
Title of host publicationPROMISE 2009 - International Conference on Predictor Models in Software Engineering
DOIs
Publication statusPublished - 1 Dec 2009
Externally publishedYes
Event5th International Conference on Predictor Models in Software Engineering, PROMISE '09 - Vancouver, BC, Canada
Duration: 18 May 200919 May 2009

Conference

Conference5th International Conference on Predictor Models in Software Engineering, PROMISE '09
CountryCanada
CityVancouver, BC
Period18/05/0919/05/09

Keywords

  • algorithms
  • experimentation
  • measurement

Cite this

Gay, G., Menzies, T., Cukic, B., & Turhan, B. (2009). How to build repeatable experiments. In PROMISE 2009 - International Conference on Predictor Models in Software Engineering [1540460] https://doi.org/10.1145/1540438.1540460