DeepMutation

mutation testing of deep learning systems

Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie, Li Li, Yang Liu, Jianjun Zhao, Yadong Wang

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

11 Citations (Scopus)

Abstract

Deep learning (DL) defines a new data-driven programming paradigm where the internal system logic is largely shaped by the training data. The standard way of evaluating DL models is to examine their performance on a test dataset. The quality of the test dataset is of great importance to gain confidence of the trained models. Using an inadequate test dataset, DL models that have achieved high test accuracy may still lack generality and robustness. In traditional software testing, mutation testing is a well-established technique for quality evaluation of test suites, which analyzes to what extent a test suite detects the injected faults. However, due to the fundamental difference between traditional software and deep learning-based software, traditional mutation testing techniques cannot be directly applied to DL systems. In this paper, we propose a mutation testing framework specialized for DL systems to measure the quality of test data. To do this, by sharing the same spirit of mutation testing in traditional software, we first define a set of source-level mutation operators to inject faults to the source of DL (i.e., training data and training programs). Then we design a set of model-level mutation operators that directly inject faults into DL models without a training process. Eventually, the quality of test data could be evaluated from the analysis on to what extent the injected faults could be detected. The usefulness of the proposed mutation testing techniques is demonstrated on two public datasets, namely MNIST and CIFAR-10, with three DL models.

Original languageEnglish
Title of host publicationProceedings - 29th IEEE International Symposium on Software Reliability Engineering, ISSRE 201818
Subtitle of host publication15–18 October 2018 Memphis, Tennessee, USA
EditorsSudipto Ghosh, Roberto Natella, Bojan Cukic, Robin Poston, Nuno Laranjeiro
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages100-111
Number of pages12
ISBN (Electronic)9781538683217
ISBN (Print)9781538683224
DOIs
Publication statusPublished - 2018
EventInternational Symposium on Software Reliability Engineering 2018 - Memphis, United States of America
Duration: 15 Oct 201818 Oct 2018
Conference number: 29th
http://2018.issre.net/

Conference

ConferenceInternational Symposium on Software Reliability Engineering 2018
Abbreviated titleISSRE 2018
CountryUnited States of America
CityMemphis
Period15/10/1818/10/18
Internet address

Keywords

  • Deep learning, Software testing, Deep neural networks, Mutation testing

Cite this

Ma, L., Zhang, F., Sun, J., Xue, M., Li, B., Juefei-Xu, F., ... Wang, Y. (2018). DeepMutation: mutation testing of deep learning systems. In S. Ghosh, R. Natella, B. Cukic, R. Poston, & N. Laranjeiro (Eds.), Proceedings - 29th IEEE International Symposium on Software Reliability Engineering, ISSRE 201818: 15–18 October 2018 Memphis, Tennessee, USA (pp. 100-111). [8539073] Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ISSRE.2018.00021
Ma, Lei ; Zhang, Fuyuan ; Sun, Jiyuan ; Xue, Minhui ; Li, Bo ; Juefei-Xu, Felix ; Xie, Chao ; Li, Li ; Liu, Yang ; Zhao, Jianjun ; Wang, Yadong. / DeepMutation : mutation testing of deep learning systems. Proceedings - 29th IEEE International Symposium on Software Reliability Engineering, ISSRE 201818: 15–18 October 2018 Memphis, Tennessee, USA. editor / Sudipto Ghosh ; Roberto Natella ; Bojan Cukic ; Robin Poston ; Nuno Laranjeiro. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2018. pp. 100-111
@inproceedings{1b1c401629fc4be5a7bc4ca7a0bb1792,
title = "DeepMutation: mutation testing of deep learning systems",
abstract = "Deep learning (DL) defines a new data-driven programming paradigm where the internal system logic is largely shaped by the training data. The standard way of evaluating DL models is to examine their performance on a test dataset. The quality of the test dataset is of great importance to gain confidence of the trained models. Using an inadequate test dataset, DL models that have achieved high test accuracy may still lack generality and robustness. In traditional software testing, mutation testing is a well-established technique for quality evaluation of test suites, which analyzes to what extent a test suite detects the injected faults. However, due to the fundamental difference between traditional software and deep learning-based software, traditional mutation testing techniques cannot be directly applied to DL systems. In this paper, we propose a mutation testing framework specialized for DL systems to measure the quality of test data. To do this, by sharing the same spirit of mutation testing in traditional software, we first define a set of source-level mutation operators to inject faults to the source of DL (i.e., training data and training programs). Then we design a set of model-level mutation operators that directly inject faults into DL models without a training process. Eventually, the quality of test data could be evaluated from the analysis on to what extent the injected faults could be detected. The usefulness of the proposed mutation testing techniques is demonstrated on two public datasets, namely MNIST and CIFAR-10, with three DL models.",
keywords = "Deep learning, Software testing, Deep neural networks, Mutation testing",
author = "Lei Ma and Fuyuan Zhang and Jiyuan Sun and Minhui Xue and Bo Li and Felix Juefei-Xu and Chao Xie and Li Li and Yang Liu and Jianjun Zhao and Yadong Wang",
year = "2018",
doi = "10.1109/ISSRE.2018.00021",
language = "English",
isbn = "9781538683224",
pages = "100--111",
editor = "Sudipto Ghosh and Roberto Natella and Bojan Cukic and Robin Poston and Nuno Laranjeiro",
booktitle = "Proceedings - 29th IEEE International Symposium on Software Reliability Engineering, ISSRE 201818",
publisher = "IEEE, Institute of Electrical and Electronics Engineers",
address = "United States of America",

}

Ma, L, Zhang, F, Sun, J, Xue, M, Li, B, Juefei-Xu, F, Xie, C, Li, L, Liu, Y, Zhao, J & Wang, Y 2018, DeepMutation: mutation testing of deep learning systems. in S Ghosh, R Natella, B Cukic, R Poston & N Laranjeiro (eds), Proceedings - 29th IEEE International Symposium on Software Reliability Engineering, ISSRE 201818: 15–18 October 2018 Memphis, Tennessee, USA., 8539073, IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 100-111, International Symposium on Software Reliability Engineering 2018, Memphis, United States of America, 15/10/18. https://doi.org/10.1109/ISSRE.2018.00021

DeepMutation : mutation testing of deep learning systems. / Ma, Lei; Zhang, Fuyuan; Sun, Jiyuan; Xue, Minhui; Li, Bo; Juefei-Xu, Felix; Xie, Chao; Li, Li; Liu, Yang; Zhao, Jianjun; Wang, Yadong.

Proceedings - 29th IEEE International Symposium on Software Reliability Engineering, ISSRE 201818: 15–18 October 2018 Memphis, Tennessee, USA. ed. / Sudipto Ghosh; Roberto Natella; Bojan Cukic; Robin Poston; Nuno Laranjeiro. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2018. p. 100-111 8539073.

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

TY - GEN

T1 - DeepMutation

T2 - mutation testing of deep learning systems

AU - Ma, Lei

AU - Zhang, Fuyuan

AU - Sun, Jiyuan

AU - Xue, Minhui

AU - Li, Bo

AU - Juefei-Xu, Felix

AU - Xie, Chao

AU - Li, Li

AU - Liu, Yang

AU - Zhao, Jianjun

AU - Wang, Yadong

PY - 2018

Y1 - 2018

N2 - Deep learning (DL) defines a new data-driven programming paradigm where the internal system logic is largely shaped by the training data. The standard way of evaluating DL models is to examine their performance on a test dataset. The quality of the test dataset is of great importance to gain confidence of the trained models. Using an inadequate test dataset, DL models that have achieved high test accuracy may still lack generality and robustness. In traditional software testing, mutation testing is a well-established technique for quality evaluation of test suites, which analyzes to what extent a test suite detects the injected faults. However, due to the fundamental difference between traditional software and deep learning-based software, traditional mutation testing techniques cannot be directly applied to DL systems. In this paper, we propose a mutation testing framework specialized for DL systems to measure the quality of test data. To do this, by sharing the same spirit of mutation testing in traditional software, we first define a set of source-level mutation operators to inject faults to the source of DL (i.e., training data and training programs). Then we design a set of model-level mutation operators that directly inject faults into DL models without a training process. Eventually, the quality of test data could be evaluated from the analysis on to what extent the injected faults could be detected. The usefulness of the proposed mutation testing techniques is demonstrated on two public datasets, namely MNIST and CIFAR-10, with three DL models.

AB - Deep learning (DL) defines a new data-driven programming paradigm where the internal system logic is largely shaped by the training data. The standard way of evaluating DL models is to examine their performance on a test dataset. The quality of the test dataset is of great importance to gain confidence of the trained models. Using an inadequate test dataset, DL models that have achieved high test accuracy may still lack generality and robustness. In traditional software testing, mutation testing is a well-established technique for quality evaluation of test suites, which analyzes to what extent a test suite detects the injected faults. However, due to the fundamental difference between traditional software and deep learning-based software, traditional mutation testing techniques cannot be directly applied to DL systems. In this paper, we propose a mutation testing framework specialized for DL systems to measure the quality of test data. To do this, by sharing the same spirit of mutation testing in traditional software, we first define a set of source-level mutation operators to inject faults to the source of DL (i.e., training data and training programs). Then we design a set of model-level mutation operators that directly inject faults into DL models without a training process. Eventually, the quality of test data could be evaluated from the analysis on to what extent the injected faults could be detected. The usefulness of the proposed mutation testing techniques is demonstrated on two public datasets, namely MNIST and CIFAR-10, with three DL models.

KW - Deep learning, Software testing, Deep neural networks, Mutation testing

UR - http://www.scopus.com/inward/record.url?scp=85056557793&partnerID=8YFLogxK

U2 - 10.1109/ISSRE.2018.00021

DO - 10.1109/ISSRE.2018.00021

M3 - Conference Paper

SN - 9781538683224

SP - 100

EP - 111

BT - Proceedings - 29th IEEE International Symposium on Software Reliability Engineering, ISSRE 201818

A2 - Ghosh, Sudipto

A2 - Natella, Roberto

A2 - Cukic, Bojan

A2 - Poston, Robin

A2 - Laranjeiro, Nuno

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

ER -

Ma L, Zhang F, Sun J, Xue M, Li B, Juefei-Xu F et al. DeepMutation: mutation testing of deep learning systems. In Ghosh S, Natella R, Cukic B, Poston R, Laranjeiro N, editors, Proceedings - 29th IEEE International Symposium on Software Reliability Engineering, ISSRE 201818: 15–18 October 2018 Memphis, Tennessee, USA. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. 2018. p. 100-111. 8539073 https://doi.org/10.1109/ISSRE.2018.00021