Unblind your apps: predicting natural-language labels for mobile gui components by deep learning

Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwe Xu, Liming Zhu, Guoqiang Li, Jinshui Wang

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

21 Citations (Scopus)

Abstract

According to the World Health Organization(WHO), it is estimated that approximately 1.3 billion people live with some forms of vision impairment globally, of whom 36 million are blind. Due to their disability, engaging these minority into the society is a challenging problem. The recent rise of smart mobile phones provides a new solution by enabling blind users' convenient access to the information and service for understanding the world. Users with vision impairment can adopt the screen reader embedded in the mobile operating systems to read the content of each screen within the app, and use gestures to interact with the phone. However, the prerequisite of using screen readers is that developers have to add natural-language labels to the image-based components when they are developing the app. Unfortunately, more than 77% apps have issues of missing labels, according to our analysis of 10,408 Android apps. Most of these issues are caused by developers' lack of awareness and knowledge in considering the minority. And even if developers want to add the labels to UI components, they may not come up with concise and clear description as most of them are of no visual issues. To overcome these challenges, we develop a deeplearning based model, called LabelDroid, to automatically predict the labels of image-based buttons by learning from large-scale commercial apps in Google Play. The experimental results show that our model can make accurate predictions and the generated labels are of higher quality than that from real Android developers.

Original languageEnglish
Title of host publicationProceedings - 2020 ACM/IEEE 42nd International Conference on Software Engineering, ICSE 2020
EditorsJane Cleland-Huang, Darko Marinov
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Pages322-334
Number of pages13
ISBN (Electronic)9781450371216
DOIs
Publication statusPublished - 2020
EventInternational Conference on Software Engineering 2020 - Virtual, Online, Korea, Republic of (South)
Duration: 27 Jun 202019 Jul 2020
Conference number: 42nd
https://dl.acm.org/doi/proceedings/10.1145/3377811 (Proceedings)
https://conf.researchr.org/home/icse-2020 (Website)

Conference

ConferenceInternational Conference on Software Engineering 2020
Abbreviated titleICSE 2020
Country/TerritoryKorea, Republic of (South)
CityVirtual, Online
Period27/06/2019/07/20
Internet address

Keywords

  • Accessibility
  • Content description
  • Image-based buttons
  • Neural networks
  • User interface

Cite this