Owl Eyes: spotting UI display issues via visual understanding

Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, Qing Wang

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

72 Citations (Scopus)

Abstract

Graphical User Interface (GUI) provides a visual bridge between a software application and end users, through which they can interact with each other. With the development of technology and aesthetics, the visual effects of the GUI are more and more attracting. However, such GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, blurred screen, missing image always occur during GUI rendering on different devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a novel approach, Owl Eye, based on deep learning for modelling visual information of the GUI screenshot. Therefore, Owl Eye can detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. We manually construct a large-scale labelled dataset with 4, 470 GUI screenshots with UI display issues and develop a heuristics-based data augmentation method for boosting the performance of our Owl Eye. The evaluation demonstrates that our Owl Eye can achieve 85% precision and 84% recall in detecting UI display issues, and 90% accuracy in localizing these issues. We also evaluate Owl Eye with popular Android apps on Google Play and F-droid, and successfully uncover 57 previously-undetected UI display issues with 26 of them being confirmed or fixed so far.

Original languageEnglish
Title of host publicationProceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020
EditorsClaire Le Goues, David Lo
Place of PublicationNew York NY USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages398-409
Number of pages12
ISBN (Electronic)9781450367684
DOIs
Publication statusPublished - 2020
Externally publishedYes
EventAutomated Software Engineering Conference 2020 - Virtual, Melbourne, Australia
Duration: 21 Sept 202025 Sept 2020
Conference number: 35th
https://dl.acm.org/doi/proceedings/10.1145/3324884 (Proceedings)
https://conf.researchr.org/home/ase-2020 (Website)
https://dl.acm.org/doi/proceedings/10.1145/3417113 (Proceedings)

Conference

ConferenceAutomated Software Engineering Conference 2020
Abbreviated titleASE 2020
Country/TerritoryAustralia
CityMelbourne
Period21/09/2025/09/20
Internet address

Keywords

  • Deep Learning
  • Mobile App
  • UI display
  • UI testing

Cite this