Learning design semantics for mobile apps

Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, Ranjitha Kumar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recently, researchers have developed black-box approaches to mine design and interaction data from mobile apps. Although the data captured during this interaction mining is descriptive, it does not expose the design semantics of UIs: what elements on the screen mean and how they are used. This paper introduces an automatic approach for generating semantic annotations for mobile app UIs. Through an iterative open coding of 73k UI elements and 720 screens, we contribute a lexical database of 25 types of UI components, 197 text button concepts, and 135 icon classes shared across apps. We use this labeled data to learn code-based patterns to detect UI components and to train a convolutional neural network that distinguishes between icon classes with 94% accuracy. To demonstrate the efficacy of our approach at scale, we compute semantic annotations for the 72k unique UIs in the Rico dataset, assigning labels for 78% of the total visible, non-redundant elements.

Original languageEnglish (US)
Title of host publicationUIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology
PublisherAssociation for Computing Machinery, Inc
Pages569-579
Number of pages11
ISBN (Electronic)9781450359481
DOIs
StatePublished - Oct 11 2018
Event31st Annual ACM Symposium on User Interface Software and Technology, UIST 2018 - Berlin, Germany
Duration: Oct 14 2018Oct 17 2018

Publication series

NameUIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

Other

Other31st Annual ACM Symposium on User Interface Software and Technology, UIST 2018
CountryGermany
CityBerlin
Period10/14/1810/17/18

    Fingerprint

Keywords

  • Design semantics
  • Machine learning
  • Mobile app design

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design
  • Software

Cite this

Liu, T. F., Craft, M., Situ, J., Yumer, E., Mech, R., & Kumar, R. (2018). Learning design semantics for mobile apps. In UIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (pp. 569-579). (UIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology). Association for Computing Machinery, Inc. https://doi.org/10.1145/3242587.3242650