From UI design image to GUI skeleton: a neural machine translator to bootstrap mobile GUI implementation

Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, Yang Liu

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

131 Citations (Scopus)


A GUI skeleton is the starting point for implementing a UI design image. To obtain a GUI skeleton from a UI design image, developers have to visually understand UI elements and their spatial layout in the image, and then translate this understanding into proper GUI components and their compositions. Automating this visual understanding and translation would be beneficial for bootstraping mobile GUI implementation, but it is a challenging task due to the diversity of UI designs and the complexity of GUI skeletons to generate. Existing tools are rigid as they depend on heuristically-designed visual understanding and GUI generation rules. In this paper, we present a neural machine translator that combines recent advances in computer vision and machine translation for translating a UI design image into a GUI skeleton. Our translator learns to extract visual features in UI images, encode these features' spatial layouts, and generate GUI skeletons in a unified neural network framework, without requiring manual rule development. For training our translator, we develop an automated GUI exploration method to automatically collect large-scale UI data from real-world applications. We carry out extensive experiments to evaluate the accuracy, generality and usefulness of our approach.

Original languageEnglish
Title of host publicationProceedings - 2018 ACM/IEEE 40th International Conference on Software Engineering, ICSE 2018
EditorsMarsha Chechik, Mark Harman
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages12
ISBN (Print)9781450356381
Publication statusPublished - 27 May 2018
Externally publishedYes
EventInternational Conference on Software Engineering 2018 - Gothenburg, Sweden
Duration: 27 May 20183 Jun 2018
Conference number: 40th (Proceedings)


ConferenceInternational Conference on Software Engineering 2018
Abbreviated titleICSE 2018
Internet address


  • Deep learning
  • Reverse engineering
  • User interface

Cite this