DeepCT

tomographic combinatorial testing for deep learning systems

Lei Ma, Felix Juefei-Xu, Minhui Xue, Bo Li, Li Li, Yang Liu, Jianjun Zhao

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

6 Citations (Scopus)

Abstract

Deep learning (DL) has achieved remarkable progress over the past decade and has been widely applied to many industry domains. However, the robustness of DL systems recently becomes great concerns, where minor perturbation on the input might cause the DL malfunction. These robustness issues could potentially result in severe consequences when a DL system is deployed to safety-critical applications and hinder the real-world deployment of DL systems. Testing techniques enable the robustness evaluation and vulnerable issue detection of a DL system at an early stage. The main challenge of testing a DL system attributes to the high dimensionality of its inputs and large internal latent feature space, which makes testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to balance the testing exploration effort and defect detection capabilities. In this paper, we perform an exploratory study of CT on DL systems. We propose a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems.

Original languageEnglish
Title of host publication2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering
EditorsXinyu Wang, David Lo, Emad Shihab
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages614-618
Number of pages5
ISBN (Electronic)9781728105918
ISBN (Print)9781728105925
DOIs
Publication statusPublished - 2019
EventIEEE International Conference on Software Analysis, Evolution, and Reengineering 2019 - Hangzhou, China
Duration: 24 Feb 201927 Feb 2019
Conference number: 26th
https://saner2019.github.io/

Conference

ConferenceIEEE International Conference on Software Analysis, Evolution, and Reengineering 2019
Abbreviated titleSANER 2019
CountryChina
CityHangzhou
Period24/02/1927/02/19
Internet address

Keywords

  • combinatorial testing
  • Deep learning
  • robustness

Cite this

Ma, L., Juefei-Xu, F., Xue, M., Li, B., Li, L., Liu, Y., & Zhao, J. (2019). DeepCT: tomographic combinatorial testing for deep learning systems. In X. Wang, D. Lo, & E. Shihab (Eds.), 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering (pp. 614-618). [8668044] Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/SANER.2019.8668044
Ma, Lei ; Juefei-Xu, Felix ; Xue, Minhui ; Li, Bo ; Li, Li ; Liu, Yang ; Zhao, Jianjun. / DeepCT : tomographic combinatorial testing for deep learning systems. 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering. editor / Xinyu Wang ; David Lo ; Emad Shihab. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2019. pp. 614-618
@inproceedings{c3ddd3eb74df4de3b94eb8beb8d577c3,
title = "DeepCT: tomographic combinatorial testing for deep learning systems",
abstract = "Deep learning (DL) has achieved remarkable progress over the past decade and has been widely applied to many industry domains. However, the robustness of DL systems recently becomes great concerns, where minor perturbation on the input might cause the DL malfunction. These robustness issues could potentially result in severe consequences when a DL system is deployed to safety-critical applications and hinder the real-world deployment of DL systems. Testing techniques enable the robustness evaluation and vulnerable issue detection of a DL system at an early stage. The main challenge of testing a DL system attributes to the high dimensionality of its inputs and large internal latent feature space, which makes testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to balance the testing exploration effort and defect detection capabilities. In this paper, we perform an exploratory study of CT on DL systems. We propose a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems.",
keywords = "combinatorial testing, Deep learning, robustness",
author = "Lei Ma and Felix Juefei-Xu and Minhui Xue and Bo Li and Li Li and Yang Liu and Jianjun Zhao",
year = "2019",
doi = "10.1109/SANER.2019.8668044",
language = "English",
isbn = "9781728105925",
pages = "614--618",
editor = "Xinyu Wang and David Lo and Emad Shihab",
booktitle = "2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering",
publisher = "IEEE, Institute of Electrical and Electronics Engineers",
address = "United States of America",

}

Ma, L, Juefei-Xu, F, Xue, M, Li, B, Li, L, Liu, Y & Zhao, J 2019, DeepCT: tomographic combinatorial testing for deep learning systems. in X Wang, D Lo & E Shihab (eds), 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering., 8668044, IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 614-618, IEEE International Conference on Software Analysis, Evolution, and Reengineering 2019, Hangzhou, China, 24/02/19. https://doi.org/10.1109/SANER.2019.8668044

DeepCT : tomographic combinatorial testing for deep learning systems. / Ma, Lei; Juefei-Xu, Felix; Xue, Minhui; Li, Bo; Li, Li; Liu, Yang; Zhao, Jianjun.

2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering. ed. / Xinyu Wang; David Lo; Emad Shihab. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2019. p. 614-618 8668044.

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

TY - GEN

T1 - DeepCT

T2 - tomographic combinatorial testing for deep learning systems

AU - Ma, Lei

AU - Juefei-Xu, Felix

AU - Xue, Minhui

AU - Li, Bo

AU - Li, Li

AU - Liu, Yang

AU - Zhao, Jianjun

PY - 2019

Y1 - 2019

N2 - Deep learning (DL) has achieved remarkable progress over the past decade and has been widely applied to many industry domains. However, the robustness of DL systems recently becomes great concerns, where minor perturbation on the input might cause the DL malfunction. These robustness issues could potentially result in severe consequences when a DL system is deployed to safety-critical applications and hinder the real-world deployment of DL systems. Testing techniques enable the robustness evaluation and vulnerable issue detection of a DL system at an early stage. The main challenge of testing a DL system attributes to the high dimensionality of its inputs and large internal latent feature space, which makes testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to balance the testing exploration effort and defect detection capabilities. In this paper, we perform an exploratory study of CT on DL systems. We propose a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems.

AB - Deep learning (DL) has achieved remarkable progress over the past decade and has been widely applied to many industry domains. However, the robustness of DL systems recently becomes great concerns, where minor perturbation on the input might cause the DL malfunction. These robustness issues could potentially result in severe consequences when a DL system is deployed to safety-critical applications and hinder the real-world deployment of DL systems. Testing techniques enable the robustness evaluation and vulnerable issue detection of a DL system at an early stage. The main challenge of testing a DL system attributes to the high dimensionality of its inputs and large internal latent feature space, which makes testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to balance the testing exploration effort and defect detection capabilities. In this paper, we perform an exploratory study of CT on DL systems. We propose a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems.

KW - combinatorial testing

KW - Deep learning

KW - robustness

UR - http://www.scopus.com/inward/record.url?scp=85064176909&partnerID=8YFLogxK

U2 - 10.1109/SANER.2019.8668044

DO - 10.1109/SANER.2019.8668044

M3 - Conference Paper

SN - 9781728105925

SP - 614

EP - 618

BT - 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering

A2 - Wang, Xinyu

A2 - Lo, David

A2 - Shihab, Emad

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

ER -

Ma L, Juefei-Xu F, Xue M, Li B, Li L, Liu Y et al. DeepCT: tomographic combinatorial testing for deep learning systems. In Wang X, Lo D, Shihab E, editors, 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. 2019. p. 614-618. 8668044 https://doi.org/10.1109/SANER.2019.8668044