Abstract
Graphical User Interface (GUI) provides a visual bridge between a software application and end users, through which they can interact with each other. With the development of technology and aesthetics, the visual effects of the GUI are more and more attracting. However, such GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, blurred screen, missing image always occur during GUI rendering on different devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a novel approach, Owl Eye, based on deep learning for modelling visual information of the GUI screenshot. Therefore, Owl Eye can detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. We manually construct a large-scale labelled dataset with 4, 470 GUI screenshots with UI display issues and develop a heuristics-based data augmentation method for boosting the performance of our Owl Eye. The evaluation demonstrates that our Owl Eye can achieve 85% precision and 84% recall in detecting UI display issues, and 90% accuracy in localizing these issues. We also evaluate Owl Eye with popular Android apps on Google Play and F-droid, and successfully uncover 57 previously-undetected UI display issues with 26 of them being confirmed or fixed so far.
Original language | English |
---|---|
Title of host publication | Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020 |
Editors | Claire Le Goues, David Lo |
Place of Publication | New York NY USA |
Publisher | IEEE, Institute of Electrical and Electronics Engineers |
Pages | 398-409 |
Number of pages | 12 |
ISBN (Electronic) | 9781450367684 |
DOIs | |
Publication status | Published - 2020 |
Externally published | Yes |
Event | Automated Software Engineering Conference 2020 - Virtual, Melbourne, Australia Duration: 21 Sept 2020 → 25 Sept 2020 Conference number: 35th https://dl.acm.org/doi/proceedings/10.1145/3324884 (Proceedings) https://conf.researchr.org/home/ase-2020 (Website) https://dl.acm.org/doi/proceedings/10.1145/3417113 (Proceedings) |
Conference
Conference | Automated Software Engineering Conference 2020 |
---|---|
Abbreviated title | ASE 2020 |
Country/Territory | Australia |
City | Melbourne |
Period | 21/09/20 → 25/09/20 |
Internet address |
|
Keywords
- Deep Learning
- Mobile App
- UI display
- UI testing