Lessons learned from a partial replication of an experiment in the context of a software engineering course

Robert Ramač, Itir Karac, Burak Turhan, Natalia Juristo, Vladimir Mandić

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearch


Replications are an integral component of experimentation through which the validity and reliability of the observed outcome in a previous experiment can be probed. In a strict replication, the experiment is executed in the same conditions as the original by following the same protocol and thus the evidence is strengthened statistically by means of increased sample size. Another objective for running replications is generalizing the experimental results beyond the limitations of one study and its context. For this purpose, certain elements of the original experiment, such as experimenters, experimental objects, and construct operationalization are altered and their impact is investigated. This paper presents lessons learned from a replication that was conducted as a part of an undergraduate university course in Serbia. The focus of the experiment was investigating the effectiveness of writing tests during the development process. The original experiment investigated the effectiveness of test-first programming and was conducted in Italy (Politecnico di Torino) with third year computer science students during an intensive Java course. Lessons learned from this partial replication are that the given task descriptions and structure has an impact on the experiment outcome and that variations in metrics collection can occur when multiple researchers analyse the data, which requires metrics consolidation.
Original languageEnglish
Title of host publicationXVII International Scientific Conference on Industrial Systems (IS'17)
Subtitle of host publicationNovi Sad, Serbia, October 4. – 6. 2017
EditorsDragutin Zelenovic, Ilija Cosic
Place of PublicationNovi Sad Serbia
PublisherUniversity of Novi Sad
Number of pages6
ISBN (Electronic)9788678929786
Publication statusPublished - 2017
Externally publishedYes
EventInternational Scientific Conference on Industrial Systems 2017 - Novi Sad, Serbia
Duration: 4 Oct 20176 Oct 2017
Conference number: 17th


ConferenceInternational Scientific Conference on Industrial Systems 2017
Abbreviated titleIS'17
CityNovi Sad
Internet address


  • software testing process
  • empirical software engineering
  • controlled experiments
  • partial replication
  • software development process

Cite this