ASDF: A differential testing framework for Automatic Speech Recognition systems

Daniel Hao Xian Yuen, Andrew Yong Chen Pang, Zhou Yang, Chun Yong Chong, Mei Kuan Lim, David Lo

Research output: Chapter in Book/Report/Conference proceedingConference PaperOther

3 Citations (Scopus)

Abstract

Recent years have witnessed wider adoption of Automated Speech Recognition (ASR) techniques in various domains. Consequently, evaluating and enhancing the quality of ASR systems is of great importance. This paper proposes Asdf, an Automated Speech Recognition Differential Testing Framework to test ASR systems. Asdf extends an existing ASR testing tool, the CrossASR++, which synthesizes test cases from a text corpus. However, CrossASR++ fails to make use of the text corpus efficiently and provides limited information on how the failed test cases can improve ASR systems. To address these limitations, our tool incorporates two novel features: (1) a text transformation module to boost the number of generated test cases and uncover more errors in ASR systems, and (2) a phonetic analysis module to identify phonemes that the ASR systems tend to transcribe incorrectly. Asdf generates more high-quality test cases by applying various text transformation methods (e.g., changing tense) to the input text in a failed test case. By doing so, Asdf can utilize a small text corpus to generate a large number of audio test cases, something which CrossASR++ is not capable of. In addition, Asdf implements more metrics to evaluate the performance of ASR systems from multiple perspectives. Asdf performs phonetic analysis on the identified failed test cases to identify the phonemes that ASR systems tend to transcribe incorrectly, providing useful information for developers to improve ASR systems. The demonstration video of our tool is made online at https://www.youtube.com/watch?v=DzVwfc3h9As. The implementation is available at https://github.com/danielyuenhx/asdf-differential-testing.

Original languageEnglish
Title of host publicationProceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation, ICST 2023
EditorsMarkus Borg, Tingting Yu
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages461-463
Number of pages3
ISBN (Electronic)9781665456661
ISBN (Print)9781665456678
DOIs
Publication statusPublished - 2023
Event16th IEEE International Conference on Software Testing, Verification and Validation, ICST 2023 - Dublin, Ireland
Duration: 16 Apr 202320 Apr 2023
Conference number: 16th
https://ieeexplore.ieee.org/xpl/conhome/10132156/proceeding (Proceedings)
https://conf.researchr.org/home/icst-2023 (Website)

Publication series

Name2023 IEEE Conference on Software Testing, Verification and Validation (ICST)
Publisher IEEE, Institute of Electrical and Electronics Engineers
ISSN (Electronic)2159-4848

Conference

Conference16th IEEE International Conference on Software Testing, Verification and Validation, ICST 2023
Abbreviated titleICST 2023
Country/TerritoryIreland
CityDublin
Period16/04/2320/04/23
Internet address

Keywords

  • automated speech recognition
  • differential testing
  • testing framework

Cite this