What leads to a confirmatory or disconfirmatory behaviour of software testers?

Iflaah Salman, Pilar Rodriguez, Burak Turhan, Ayse Tosun, Arda Gureller

Research output: Contribution to journalArticleResearchpeer-review


Background: The existing literature in software engineering reports adverse effects of confirmation bias on software testing. Confirmation bias among software testers leads to confirmatory behaviour, which is designing or executing relatively more specification consistent test cases (confirmatory behaviour) than specification inconsistent test cases (disconfirmatory behaviour). Objective: We aim to explore the antecedents to confirmatory and disconfirmatory behaviour of software testers. Furthermore, we aim to understand why and how those antecedents lead to (dis)confirmatory behaviour. Method: We follow grounded theory method for the analyses of the data collected through semi-structured interviews with twelve software testers. Results: We identified twenty antecedents to (dis)confirmatory behaviour, and classified them in nine categories. Experience and Time are the two major categories. Experience is a disconfirmatory category, which also determines which behaviour (confirmatory or disconfirmatory) occurs first among software testers, as an effect of other antecedents. Time Pressure is a confirmatory antecedent of the Time category. It also contributes to the confirmatory effects of antecedents of other categories. Conclusion: The disconfirmatory antecedents, especially that belong to the testing process, e.g., test suite reviews by project team members, may help circumvent the deleterious effects of confirmation bias in software testing. If a team's resources permit, the designing and execution of a test suite could be divided among the test team members, as different perspectives of testers may help to detect more errors. The results of our study are based on a single context where dedicated testing teams focus on higher levels of testing. The study's scope does not account for the testing performed by developers. Future work includes exploring other contexts to extend our results.
Original languageEnglish
Number of pages18
JournalIEEE Transactions on Software Engineering
Publication statusAccepted/In press - 27 Aug 2020


  • Software Testing
  • Cognitive Biases
  • Confirmation Bias
  • Grounded Theory
  • Interviews

Cite this