Learning design semantics for mobile apps

Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, Ranjitha Kumar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recently, researchers have developed black-box approaches to mine design and interaction data from mobile apps. Although the data captured during this interaction mining is descriptive, it does not expose the design semantics of UIs: what elements on the screen mean and how they are used. This paper introduces an automatic approach for generating semantic annotations for mobile app UIs. Through an iterative open coding of 73k UI elements and 720 screens, we contribute a lexical database of 25 types of UI components, 197 text button concepts, and 135 icon classes shared across apps. We use this labeled data to learn code-based patterns to detect UI components and to train a convolutional neural network that distinguishes between icon classes with 94% accuracy. To demonstrate the efficacy of our approach at scale, we compute semantic annotations for the 72k unique UIs in the Rico dataset, assigning labels for 78% of the total visible, non-redundant elements.

Original languageEnglish (US)
Title of host publicationUIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology
PublisherAssociation for Computing Machinery, Inc
Pages569-579
Number of pages11
ISBN (Electronic)9781450359481
DOIs
StatePublished - Oct 11 2018
Event31st Annual ACM Symposium on User Interface Software and Technology, UIST 2018 - Berlin, Germany
Duration: Oct 14 2018Oct 17 2018

Other

Other31st Annual ACM Symposium on User Interface Software and Technology, UIST 2018
CountryGermany
CityBerlin
Period10/14/1810/17/18

Fingerprint

Application programs
Semantics
Labels
Neural networks

Keywords

  • Design semantics
  • Machine learning
  • Mobile app design

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design
  • Software

Cite this

Liu, T. F., Craft, M., Situ, J., Yumer, E., Mech, R., & Kumar, R. (2018). Learning design semantics for mobile apps. In UIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (pp. 569-579). Association for Computing Machinery, Inc. https://doi.org/10.1145/3242587.3242650

Learning design semantics for mobile apps. / Liu, Thomas F.; Craft, Mark; Situ, Jason; Yumer, Ersin; Mech, Radomir; Kumar, Ranjitha.

UIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, Inc, 2018. p. 569-579.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Liu, TF, Craft, M, Situ, J, Yumer, E, Mech, R & Kumar, R 2018, Learning design semantics for mobile apps. in UIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, Inc, pp. 569-579, 31st Annual ACM Symposium on User Interface Software and Technology, UIST 2018, Berlin, Germany, 10/14/18. https://doi.org/10.1145/3242587.3242650
Liu TF, Craft M, Situ J, Yumer E, Mech R, Kumar R. Learning design semantics for mobile apps. In UIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, Inc. 2018. p. 569-579 https://doi.org/10.1145/3242587.3242650
Liu, Thomas F. ; Craft, Mark ; Situ, Jason ; Yumer, Ersin ; Mech, Radomir ; Kumar, Ranjitha. / Learning design semantics for mobile apps. UIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, Inc, 2018. pp. 569-579
@inproceedings{8d14fd0b19964efabc03b3a531cd2c48,
title = "Learning design semantics for mobile apps",
abstract = "Recently, researchers have developed black-box approaches to mine design and interaction data from mobile apps. Although the data captured during this interaction mining is descriptive, it does not expose the design semantics of UIs: what elements on the screen mean and how they are used. This paper introduces an automatic approach for generating semantic annotations for mobile app UIs. Through an iterative open coding of 73k UI elements and 720 screens, we contribute a lexical database of 25 types of UI components, 197 text button concepts, and 135 icon classes shared across apps. We use this labeled data to learn code-based patterns to detect UI components and to train a convolutional neural network that distinguishes between icon classes with 94{\%} accuracy. To demonstrate the efficacy of our approach at scale, we compute semantic annotations for the 72k unique UIs in the Rico dataset, assigning labels for 78{\%} of the total visible, non-redundant elements.",
keywords = "Design semantics, Machine learning, Mobile app design",
author = "Liu, {Thomas F.} and Mark Craft and Jason Situ and Ersin Yumer and Radomir Mech and Ranjitha Kumar",
year = "2018",
month = "10",
day = "11",
doi = "10.1145/3242587.3242650",
language = "English (US)",
pages = "569--579",
booktitle = "UIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology",
publisher = "Association for Computing Machinery, Inc",

}

TY - GEN

T1 - Learning design semantics for mobile apps

AU - Liu, Thomas F.

AU - Craft, Mark

AU - Situ, Jason

AU - Yumer, Ersin

AU - Mech, Radomir

AU - Kumar, Ranjitha

PY - 2018/10/11

Y1 - 2018/10/11

N2 - Recently, researchers have developed black-box approaches to mine design and interaction data from mobile apps. Although the data captured during this interaction mining is descriptive, it does not expose the design semantics of UIs: what elements on the screen mean and how they are used. This paper introduces an automatic approach for generating semantic annotations for mobile app UIs. Through an iterative open coding of 73k UI elements and 720 screens, we contribute a lexical database of 25 types of UI components, 197 text button concepts, and 135 icon classes shared across apps. We use this labeled data to learn code-based patterns to detect UI components and to train a convolutional neural network that distinguishes between icon classes with 94% accuracy. To demonstrate the efficacy of our approach at scale, we compute semantic annotations for the 72k unique UIs in the Rico dataset, assigning labels for 78% of the total visible, non-redundant elements.

AB - Recently, researchers have developed black-box approaches to mine design and interaction data from mobile apps. Although the data captured during this interaction mining is descriptive, it does not expose the design semantics of UIs: what elements on the screen mean and how they are used. This paper introduces an automatic approach for generating semantic annotations for mobile app UIs. Through an iterative open coding of 73k UI elements and 720 screens, we contribute a lexical database of 25 types of UI components, 197 text button concepts, and 135 icon classes shared across apps. We use this labeled data to learn code-based patterns to detect UI components and to train a convolutional neural network that distinguishes between icon classes with 94% accuracy. To demonstrate the efficacy of our approach at scale, we compute semantic annotations for the 72k unique UIs in the Rico dataset, assigning labels for 78% of the total visible, non-redundant elements.

KW - Design semantics

KW - Machine learning

KW - Mobile app design

UR - http://www.scopus.com/inward/record.url?scp=85056909475&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85056909475&partnerID=8YFLogxK

U2 - 10.1145/3242587.3242650

DO - 10.1145/3242587.3242650

M3 - Conference contribution

SP - 569

EP - 579

BT - UIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

PB - Association for Computing Machinery, Inc

ER -