TY - GEN
T1 - App-Based Task Shortcuts for Virtual Assistants
AU - Arsan, Deniz
AU - Zaidi, Ali
AU - Sagar, Aravind
AU - Kumar, Ranjitha
N1 - Publisher Copyright:
© 2021 ACM.
PY - 2021/10/10
Y1 - 2021/10/10
N2 - Virtual assistants like Google Assistant and Siri often interface with external apps when they cannot directly perform a task. Currently, developers must manually expose the capabilities of their apps to virtual assistants, using App Actions on Android or Shortcuts on iOS. This paper presents savant, a system that automatically generates task shortcuts for virtual assistants by mapping user tasks to relevant UI screens in apps. For a given natural language task (e.g., "send money to Joe"), savant leverages text and semantic information contained within UIs to identify relevant screens, and intent modeling to parse and map entities (e.g., "Joe") to required UI inputs. Therefore, savant allows virtual assistants to interface with apps and handle new tasks without requiring any developer effort. To evaluate savant, we performed a user study to identify common tasks users perform with virtual assistants. We then demonstrate that savant can find relevant app screens for those tasks and autocomplete the UI inputs.
AB - Virtual assistants like Google Assistant and Siri often interface with external apps when they cannot directly perform a task. Currently, developers must manually expose the capabilities of their apps to virtual assistants, using App Actions on Android or Shortcuts on iOS. This paper presents savant, a system that automatically generates task shortcuts for virtual assistants by mapping user tasks to relevant UI screens in apps. For a given natural language task (e.g., "send money to Joe"), savant leverages text and semantic information contained within UIs to identify relevant screens, and intent modeling to parse and map entities (e.g., "Joe") to required UI inputs. Therefore, savant allows virtual assistants to interface with apps and handle new tasks without requiring any developer effort. To evaluate savant, we performed a user study to identify common tasks users perform with virtual assistants. We then demonstrate that savant can find relevant app screens for those tasks and autocomplete the UI inputs.
KW - Virtual assistants
KW - interaction mining
KW - mobile apps
KW - task shortcuts
UR - http://www.scopus.com/inward/record.url?scp=85118213530&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85118213530&partnerID=8YFLogxK
U2 - 10.1145/3472749.3474808
DO - 10.1145/3472749.3474808
M3 - Conference contribution
AN - SCOPUS:85118213530
T3 - UIST 2021 - Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology
SP - 1089
EP - 1099
BT - UIST 2021 - Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology
PB - Association for Computing Machinery
T2 - 34th Annual ACM Symposium on User Interface Software and Technology, UIST 2021
Y2 - 10 October 2021 through 14 October 2021
ER -