A metamorphic testing framework and toolkit for modular automated driving systems

Riley Underwood, Quang-Hung Luu, Huai Liu

Research output: Chapter in Book/Report/Conference proceedingConference PaperOther


Autonomous vehicles (AV), at their highest potentials, will provide a greater mobility, an increased traffic efficiency and, more importantly, safer trips for millions of people every day. While assuring their safety and reliability is, thus, of great importance, it is also a huge challenge. Metamorphic testing (MT) has been shown to be one of the most successful testing techniques to validate automated driving systems (ADS) underpinning the AV. Having said that, there are still lots of rooms for further improving the ADS testing with MT. On one hand, the non-determinism in ADS' behaviors poses great challenges for precisely judging their correctness. On the other hand, the testing scenarios used in the existing studies are still not very much complex for mimicking various realistic traffic conditions. In this study, we propose a new framework which takes into account the hypothesis testing to provide a more solid way for judging the non-deterministic behaviors of test outcomes. On top of that, we develop a new toolkit to implement more complex and realistic ADS testing scenarios. To demonstrate its practicability, we design complex traffic scenarios and pay attention to examine the ADS' behaviors in non-collision cases which are often unable to be detected by conventional testing methods. It is then applied to test Autoware, a state-of-the-art modular ADS using the Carla simulator. An analysis of results with the Mann-Whitney-Wilcoxon test and Cohen's d values reveals a large number of consistencies and reliability issues of Autoware. The findings highlight the flexibility and capability of our MT-based framework in validating the AV using a nondeterministic measure and realistic scenarios that can work in the absence of ground truth datasets.

Original languageEnglish
Title of host publicationProceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages8
ISBN (Electronic)9798350301762
Publication statusPublished - 2023
EventIEEE/ACM International Workshop on Metamorphic Testing 2023 - Melbourne, Australia
Duration: 14 May 202314 May 2023
Conference number: 8th
https://ieeexplore.ieee.org/xpl/conhome/10190355/proceeding (Proceedings)
http://metwiki.net/MET23/ (Website)


ConferenceIEEE/ACM International Workshop on Metamorphic Testing 2023
Abbreviated titleMET 2023
Internet address


  • autonomous vehicles
  • diversity of traffic scenarios
  • Metamorphic testing
  • self-driving cars

Cite this