Abstract
Recent years have witnessed wider adoption of Automated Speech Recognition (ASR) techniques in various domains. Consequently, evaluating and enhancing the quality of ASR systems is of great importance. This paper proposes Asdf, an Automated Speech Recognition Differential Testing Framework to test ASR systems. Asdf extends an existing ASR testing tool, the CrossASR++, which synthesizes test cases from a text corpus. However, CrossASR++ fails to make use of the text corpus efficiently and provides limited information on how the failed test cases can improve ASR systems. To address these limitations, our tool incorporates two novel features: (1) a text transformation module to boost the number of generated test cases and uncover more errors in ASR systems, and (2) a phonetic analysis module to identify phonemes that the ASR systems tend to transcribe incorrectly. Asdf generates more high-quality test cases by applying various text transformation methods (e.g., changing tense) to the input text in a failed test case. By doing so, Asdf can utilize a small text corpus to generate a large number of audio test cases, something which CrossASR++ is not capable of. In addition, Asdf implements more metrics to evaluate the performance of ASR systems from multiple perspectives. Asdf performs phonetic analysis on the identified failed test cases to identify the phonemes that ASR systems tend to transcribe incorrectly, providing useful information for developers to improve ASR systems. The demonstration video of our tool is made online at https://www.youtube.com/watch?v=DzVwfc3h9As. The implementation is available at https://github.com/danielyuenhx/asdf-differential-testing.
Original language | English |
---|---|
Title of host publication | Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation, ICST 2023 |
Editors | Markus Borg, Tingting Yu |
Publisher | IEEE, Institute of Electrical and Electronics Engineers |
Pages | 461-463 |
Number of pages | 3 |
ISBN (Electronic) | 9781665456661 |
ISBN (Print) | 9781665456678 |
DOIs | |
Publication status | Published - 2023 |
Event | 16th IEEE International Conference on Software Testing, Verification and Validation, ICST 2023 - Dublin, Ireland Duration: 16 Apr 2023 → 20 Apr 2023 Conference number: 16th https://ieeexplore.ieee.org/xpl/conhome/10132156/proceeding (Proceedings) https://conf.researchr.org/home/icst-2023 (Website) |
Publication series
Name | 2023 IEEE Conference on Software Testing, Verification and Validation (ICST) |
---|---|
Publisher | IEEE, Institute of Electrical and Electronics Engineers |
ISSN (Electronic) | 2159-4848 |
Conference
Conference | 16th IEEE International Conference on Software Testing, Verification and Validation, ICST 2023 |
---|---|
Abbreviated title | ICST 2023 |
Country/Territory | Ireland |
City | Dublin |
Period | 16/04/23 → 20/04/23 |
Internet address |
Keywords
- automated speech recognition
- differential testing
- testing framework