Seenomaly: vision-based linting of GUI animation effects against design-don't guidelines

Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, Guoqiang Li, Jinshui Wang

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

40 Citations (Scopus)


GUI animations, such as card movement, menu slide in/out, snackbar display, provide appealing user experience and enhance the usability of mobile applications. These GUI animations should not violate the platform's UI design guidelines (referred to as designdon't guideline in this work) regarding component motion and interaction, content appearing and disappearing, and elevation and shadow changes. However, none of existing static code analysis, functional GUI testing and GUI image comparison techniques can see the GUI animations on the scree, and thus they cannot support the linting of GUI animations against design-don't guidelines. In this work, we formulate this GUI animation linting problem as a multi-class screencast classification task, but we do not have sufficient labeled GUI animations to train the classifier. Instead, we propose an unsupervised, computer-vision based adversarial autoencoder to solve this linting problem. Our autoencoder learns to group similar GUI animations by seeing lots of unlabeled realapplication GUI animations and learning to generate them. As the first work of its kind, we build the datasets of synthetic and realworld GUI animations. Through experiments on these datasets, we systematically investigate the learning capability of our model and its effectiveness and practicality for linting GUI animations, and identify the challenges in this linting problem for future work.

Original languageEnglish
Title of host publicationProceedings - 2020 ACM/IEEE 42nd International Conference on Software Engineering, ICSE 2020
EditorsJane Cleland-Huang, Darko Marinov
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages12
ISBN (Electronic)9781450371216
Publication statusPublished - 2020
EventInternational Conference on Software Engineering 2020 - Online, Seoul, Korea, South
Duration: 27 Jun 202019 Jul 2020
Conference number: 42nd (Proceedings) (Website)


ConferenceInternational Conference on Software Engineering 2020
Abbreviated titleICSE 2020
Country/TerritoryKorea, South
Internet address


  • Design guidelines
  • Gui animation
  • Lint
  • Unsupervised learning

Cite this