Assurances in software testing

a roadmap

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

1 Citation (Scopus)

Abstract

As researchers, we already understand how to make testing more effective and efficient at finding bugs. However, as fuzzing (i.e., automated testing) becomes more widely adopted in practice, practitioners are asking: Which assurances does a fuzzing campaign provide that exposes no bugs? When is it safe to stop the fuzzer with a reasonable residual risk? How much longer should the fuzzer be run to achieve sufficient coverage? It is time for us to move beyond the innovation of increasingly sophisticated testing techniques, to build a body of knowledge around the explication and quantification of the testing process, and to develop sound methodologies to estimate and extrapolate these quantities with measurable accuracy. In our vision of the future practitioners leverage a rich statistical toolset to assess residual risk, to obtain statistical guarantees, and to analyze the cost-benefit trade-off for ongoing fuzzing campaigns. We propose a general framework as a first starting point to tackle this fundamental challenge and discuss a large number of concrete opportunities for future research.

Original languageEnglish
Title of host publicationProceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering
Subtitle of host publicationNew Ideas and Emerging Results, ICSE-NIER 2019
EditorsAnita Sarma, Leonarado Murta
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages5-8
Number of pages4
ISBN (Electronic)9781728117584
ISBN (Print)9781728117591
DOIs
Publication statusPublished - 2019
EventIEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results 2019 - Montreal, Canada
Duration: 25 May 201931 May 2019
Conference number: 41st
https://2019.icse-conferences.org/track/icse-2019-New-Ideas-and-Emerging-Reults?track=ICSE%20New%20Ideas%20and%20Emerging%20Results

Conference

ConferenceIEEE/ACM International Conference on Software Engineering
Abbreviated titleICSE-NIER 2019
CountryCanada
CityMontreal
Period25/05/1931/05/19
Internet address

Keywords

  • Cost benefit tradeoff
  • Guarantees
  • Residual risk
  • Statistics

Cite this

Bohme, M. (2019). Assurances in software testing: a roadmap. In A. Sarma, & L. Murta (Eds.), Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-NIER 2019 (pp. 5-8). [8805701] Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICSE-NIER.2019.00010
Bohme, Marcel. / Assurances in software testing : a roadmap. Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-NIER 2019. editor / Anita Sarma ; Leonarado Murta. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2019. pp. 5-8
@inproceedings{fdab9b26a63f4eaab550174bb9d5a2c1,
title = "Assurances in software testing: a roadmap",
abstract = "As researchers, we already understand how to make testing more effective and efficient at finding bugs. However, as fuzzing (i.e., automated testing) becomes more widely adopted in practice, practitioners are asking: Which assurances does a fuzzing campaign provide that exposes no bugs? When is it safe to stop the fuzzer with a reasonable residual risk? How much longer should the fuzzer be run to achieve sufficient coverage? It is time for us to move beyond the innovation of increasingly sophisticated testing techniques, to build a body of knowledge around the explication and quantification of the testing process, and to develop sound methodologies to estimate and extrapolate these quantities with measurable accuracy. In our vision of the future practitioners leverage a rich statistical toolset to assess residual risk, to obtain statistical guarantees, and to analyze the cost-benefit trade-off for ongoing fuzzing campaigns. We propose a general framework as a first starting point to tackle this fundamental challenge and discuss a large number of concrete opportunities for future research.",
keywords = "Cost benefit tradeoff, Guarantees, Residual risk, Statistics",
author = "Marcel Bohme",
year = "2019",
doi = "10.1109/ICSE-NIER.2019.00010",
language = "English",
isbn = "9781728117591",
pages = "5--8",
editor = "Anita Sarma and Leonarado Murta",
booktitle = "Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering",
publisher = "IEEE, Institute of Electrical and Electronics Engineers",
address = "United States of America",

}

Bohme, M 2019, Assurances in software testing: a roadmap. in A Sarma & L Murta (eds), Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-NIER 2019., 8805701, IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 5-8, IEEE/ACM International Conference on Software Engineering, Montreal, Canada, 25/05/19. https://doi.org/10.1109/ICSE-NIER.2019.00010

Assurances in software testing : a roadmap. / Bohme, Marcel.

Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-NIER 2019. ed. / Anita Sarma; Leonarado Murta. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2019. p. 5-8 8805701.

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

TY - GEN

T1 - Assurances in software testing

T2 - a roadmap

AU - Bohme, Marcel

PY - 2019

Y1 - 2019

N2 - As researchers, we already understand how to make testing more effective and efficient at finding bugs. However, as fuzzing (i.e., automated testing) becomes more widely adopted in practice, practitioners are asking: Which assurances does a fuzzing campaign provide that exposes no bugs? When is it safe to stop the fuzzer with a reasonable residual risk? How much longer should the fuzzer be run to achieve sufficient coverage? It is time for us to move beyond the innovation of increasingly sophisticated testing techniques, to build a body of knowledge around the explication and quantification of the testing process, and to develop sound methodologies to estimate and extrapolate these quantities with measurable accuracy. In our vision of the future practitioners leverage a rich statistical toolset to assess residual risk, to obtain statistical guarantees, and to analyze the cost-benefit trade-off for ongoing fuzzing campaigns. We propose a general framework as a first starting point to tackle this fundamental challenge and discuss a large number of concrete opportunities for future research.

AB - As researchers, we already understand how to make testing more effective and efficient at finding bugs. However, as fuzzing (i.e., automated testing) becomes more widely adopted in practice, practitioners are asking: Which assurances does a fuzzing campaign provide that exposes no bugs? When is it safe to stop the fuzzer with a reasonable residual risk? How much longer should the fuzzer be run to achieve sufficient coverage? It is time for us to move beyond the innovation of increasingly sophisticated testing techniques, to build a body of knowledge around the explication and quantification of the testing process, and to develop sound methodologies to estimate and extrapolate these quantities with measurable accuracy. In our vision of the future practitioners leverage a rich statistical toolset to assess residual risk, to obtain statistical guarantees, and to analyze the cost-benefit trade-off for ongoing fuzzing campaigns. We propose a general framework as a first starting point to tackle this fundamental challenge and discuss a large number of concrete opportunities for future research.

KW - Cost benefit tradeoff

KW - Guarantees

KW - Residual risk

KW - Statistics

UR - http://www.scopus.com/inward/record.url?scp=85071955508&partnerID=8YFLogxK

U2 - 10.1109/ICSE-NIER.2019.00010

DO - 10.1109/ICSE-NIER.2019.00010

M3 - Conference Paper

SN - 9781728117591

SP - 5

EP - 8

BT - Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering

A2 - Sarma, Anita

A2 - Murta, Leonarado

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

ER -

Bohme M. Assurances in software testing: a roadmap. In Sarma A, Murta L, editors, Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-NIER 2019. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. 2019. p. 5-8. 8805701 https://doi.org/10.1109/ICSE-NIER.2019.00010