Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChat

Sidong Feng, Haochuan Lu, Jianqin Jiang, Ting Xiong, Likun Huang, Yinglin Liang, Xiaoqin Li, Yuetang Deng, Aldeida Aleti

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

1 Citation (Scopus)

Abstract

UI automation tests play a crucial role in ensuring the quality of mobile applications. Despite the growing popularity of machine learning techniques to generate these tests, they still face several challenges, such as the mismatch of UI elements. The recent advances in Large Language Models (LLMs) have addressed these issues by leveraging their semantic understanding capabilities. However, a significant gap remains in applying these models to industrial-level app testing, particularly in terms of cost optimization and knowledge limitation. To address this, we introduce CAT to create cost-effective UI automation tests for industry apps by combining machine learning and LLMs with best practices. Given the task description, CAT employs Retrieval Augmented Generation (RAG) to source examples of industrial app usage as the few-shot learning context, assisting LLMs in generating the specific sequence of actions. CAT then employs machine learning techniques, with LLMs serving as a complementary optimizer, to map the target element on the UI screen. Our evaluations on the WeChat testing dataset demonstrate the CAT's performance and cost-effectiveness, achieving 90% UI automation with $0.34 cost, outperforming the state-of-the-art. We have also integrated our approach into the real-world WeChat testing platform, demonstrating its usefulness in detecting 141 bugs and enhancing the developers' testing process.

Original languageEnglish
Title of host publicationProceedings - 2024 39th ACM/IEEE International Conference on Automated Software Engineering, ASE 2024
EditorsLin Shi
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Pages1973-1978
Number of pages6
ISBN (Electronic)9798400712487
DOIs
Publication statusPublished - 2024
EventAutomated Software Engineering Conference 2024 - Sacramento, United States of America
Duration: 28 Oct 20241 Nov 2024
Conference number: 39th
https://dl.acm.org/doi/proceedings/10.1145/3691620 (Proceedings)
https://conf.researchr.org/home/ase-2024 (Website)

Conference

ConferenceAutomated Software Engineering Conference 2024
Abbreviated titleASE 2024
Country/TerritoryUnited States of America
CitySacramento
Period28/10/241/11/24
OtherACM/IEEE International Conference on Automated Software Engineering, ASE 2024
Internet address

Keywords

  • cost optimization
  • large language model
  • retrieval-augmented generation
  • UI automation test

Cite this