A controlled experiment on time pressure and confirmation bias in functional software testing

Iflaah Salman, Burak Turhan, Sira Vegas

Research output: Contribution to journalArticleResearchpeer-review

Abstract

Context: Confirmation bias is a person’s tendency to look for evidence that strengthens his/her prior beliefs rather than refutes them. Manifestation of confirmation bias in software testing may have adverse effects on software quality. Psychology research suggests that time pressure could trigger confirmation bias. Objective: In the software industry, this phenomenon may deteriorate software quality. In this study, we investigate whether testers manifest confirmation bias and how it is affected by time pressure in functional software testing. Method: We performed a controlled experiment with 42 graduate students to assess manifestation of confirmation bias in terms of the conformity of their designed test cases to the provided requirements specification. We employed a one factor with two treatments between-subjects experimental design. Results: We observed, overall, participants designed significantly more confirmatory test cases as compared to disconfirmatory ones, which is in line with previous research. However, we did not observe time pressure as an antecedent to an increased rate of confirmatory testing behaviour. Conclusion: People tend to design confirmatory test cases regardless of time pressure. For practice, we find it necessary that testers develop self-awareness of confirmation bias and counter its potential adverse effects with a disconfirmatory attitude. We recommend further replications to investigate the effect of time pressure as a potential contributor to the manifestation of confirmation bias.

Original languageEnglish
Pages (from-to)1727-1761
Number of pages35
JournalEmpirical Software Engineering
Volume24
Issue number4
DOIs
Publication statusPublished - Aug 2019

Keywords

  • Cognitive biases
  • Confirmation bias
  • Experiment
  • Human factors
  • Software testing
  • Test quality
  • Time pressure

Cite this

@article{b6c4957d2d3f4a2d81c7e07dc571169a,
title = "A controlled experiment on time pressure and confirmation bias in functional software testing",
abstract = "Context: Confirmation bias is a person’s tendency to look for evidence that strengthens his/her prior beliefs rather than refutes them. Manifestation of confirmation bias in software testing may have adverse effects on software quality. Psychology research suggests that time pressure could trigger confirmation bias. Objective: In the software industry, this phenomenon may deteriorate software quality. In this study, we investigate whether testers manifest confirmation bias and how it is affected by time pressure in functional software testing. Method: We performed a controlled experiment with 42 graduate students to assess manifestation of confirmation bias in terms of the conformity of their designed test cases to the provided requirements specification. We employed a one factor with two treatments between-subjects experimental design. Results: We observed, overall, participants designed significantly more confirmatory test cases as compared to disconfirmatory ones, which is in line with previous research. However, we did not observe time pressure as an antecedent to an increased rate of confirmatory testing behaviour. Conclusion: People tend to design confirmatory test cases regardless of time pressure. For practice, we find it necessary that testers develop self-awareness of confirmation bias and counter its potential adverse effects with a disconfirmatory attitude. We recommend further replications to investigate the effect of time pressure as a potential contributor to the manifestation of confirmation bias.",
keywords = "Cognitive biases, Confirmation bias, Experiment, Human factors, Software testing, Test quality, Time pressure",
author = "Iflaah Salman and Burak Turhan and Sira Vegas",
year = "2019",
month = "8",
doi = "10.1007/s10664-018-9668-8",
language = "English",
volume = "24",
pages = "1727--1761",
journal = "Empirical Software Engineering",
issn = "1382-3256",
publisher = "Springer-Verlag London Ltd.",
number = "4",

}

A controlled experiment on time pressure and confirmation bias in functional software testing. / Salman, Iflaah; Turhan, Burak; Vegas, Sira.

In: Empirical Software Engineering, Vol. 24, No. 4, 08.2019, p. 1727-1761.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - A controlled experiment on time pressure and confirmation bias in functional software testing

AU - Salman, Iflaah

AU - Turhan, Burak

AU - Vegas, Sira

PY - 2019/8

Y1 - 2019/8

N2 - Context: Confirmation bias is a person’s tendency to look for evidence that strengthens his/her prior beliefs rather than refutes them. Manifestation of confirmation bias in software testing may have adverse effects on software quality. Psychology research suggests that time pressure could trigger confirmation bias. Objective: In the software industry, this phenomenon may deteriorate software quality. In this study, we investigate whether testers manifest confirmation bias and how it is affected by time pressure in functional software testing. Method: We performed a controlled experiment with 42 graduate students to assess manifestation of confirmation bias in terms of the conformity of their designed test cases to the provided requirements specification. We employed a one factor with two treatments between-subjects experimental design. Results: We observed, overall, participants designed significantly more confirmatory test cases as compared to disconfirmatory ones, which is in line with previous research. However, we did not observe time pressure as an antecedent to an increased rate of confirmatory testing behaviour. Conclusion: People tend to design confirmatory test cases regardless of time pressure. For practice, we find it necessary that testers develop self-awareness of confirmation bias and counter its potential adverse effects with a disconfirmatory attitude. We recommend further replications to investigate the effect of time pressure as a potential contributor to the manifestation of confirmation bias.

AB - Context: Confirmation bias is a person’s tendency to look for evidence that strengthens his/her prior beliefs rather than refutes them. Manifestation of confirmation bias in software testing may have adverse effects on software quality. Psychology research suggests that time pressure could trigger confirmation bias. Objective: In the software industry, this phenomenon may deteriorate software quality. In this study, we investigate whether testers manifest confirmation bias and how it is affected by time pressure in functional software testing. Method: We performed a controlled experiment with 42 graduate students to assess manifestation of confirmation bias in terms of the conformity of their designed test cases to the provided requirements specification. We employed a one factor with two treatments between-subjects experimental design. Results: We observed, overall, participants designed significantly more confirmatory test cases as compared to disconfirmatory ones, which is in line with previous research. However, we did not observe time pressure as an antecedent to an increased rate of confirmatory testing behaviour. Conclusion: People tend to design confirmatory test cases regardless of time pressure. For practice, we find it necessary that testers develop self-awareness of confirmation bias and counter its potential adverse effects with a disconfirmatory attitude. We recommend further replications to investigate the effect of time pressure as a potential contributor to the manifestation of confirmation bias.

KW - Cognitive biases

KW - Confirmation bias

KW - Experiment

KW - Human factors

KW - Software testing

KW - Test quality

KW - Time pressure

UR - http://www.scopus.com/inward/record.url?scp=85069680730&partnerID=8YFLogxK

U2 - 10.1007/s10664-018-9668-8

DO - 10.1007/s10664-018-9668-8

M3 - Article

VL - 24

SP - 1727

EP - 1761

JO - Empirical Software Engineering

JF - Empirical Software Engineering

SN - 1382-3256

IS - 4

ER -