TY - GEN
T1 - LLM-powered Multimodal Insight Summarization for UX Testing
AU - Turbeville, Kelsey
AU - Muengtaweepongsa, Jennarong
AU - Stevens, Samuel
AU - Moss, Jason
AU - Pon, Amy
AU - Lee, Kyra
AU - Mehra, Charu
AU - Villalobos, Jenny Gutierrez
AU - Kumar, Ranjitha
N1 - The authors would like to thank the reviewers for their helpful comments and suggestions. We would like to thank Jerry O. Talton for his multimodal insights; and Kaj van de Loo and Michelle Engle for their technical and product guidance. Finally, we would like to thank the members of the Data Platform and Machine Learning team at UserTesting and Scott Hutchins for their eforts in implementing and deploying this work in a real system.
PY - 2024/11/4
Y1 - 2024/11/4
N2 - User experience (UX) testing platforms capture many data types related to user feedback and behavior, including clickstream, survey responses, screen recordings of participants performing tasks, and participants’ think-aloud audio. Analyzing these multimodal data channels to extract insights remains a time-consuming, manual process for UX researchers. This paper presents a large language model (LLM) approach for generating insights from multimodal UX testing data. By unifying verbal, behavioral, and design data streams into a novel natural language representation, we construct LLM prompts that generate insights combining information across all data types. Each insight can be traced back to behavioral and verbal evidence, allowing users to quickly verify accuracy. We evaluate LLM-generated insight summaries by deploying them in a popular remote UX testing platform, and present evidence that they help UX researchers more efciently identify key fndings from UX tests.
AB - User experience (UX) testing platforms capture many data types related to user feedback and behavior, including clickstream, survey responses, screen recordings of participants performing tasks, and participants’ think-aloud audio. Analyzing these multimodal data channels to extract insights remains a time-consuming, manual process for UX researchers. This paper presents a large language model (LLM) approach for generating insights from multimodal UX testing data. By unifying verbal, behavioral, and design data streams into a novel natural language representation, we construct LLM prompts that generate insights combining information across all data types. Each insight can be traced back to behavioral and verbal evidence, allowing users to quickly verify accuracy. We evaluate LLM-generated insight summaries by deploying them in a popular remote UX testing platform, and present evidence that they help UX researchers more efciently identify key fndings from UX tests.
KW - UX research
KW - large language models
KW - multimodal insights
KW - usability testing
UR - http://www.scopus.com/inward/record.url?scp=85212592069&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85212592069&partnerID=8YFLogxK
U2 - 10.1145/3678957.3685701
DO - 10.1145/3678957.3685701
M3 - Conference contribution
AN - SCOPUS:85212592069
T3 - ACM International Conference Proceeding Series
SP - 4
EP - 11
BT - ICMI 2024 - Proceedings of the 26th International Conference on Multimodal Interaction
PB - Association for Computing Machinery
T2 - 26th International Conference on Multimodal Interaction, ICMI 2024
Y2 - 4 November 2024 through 8 November 2024
ER -