TY - JOUR
T1 - Can Computers Outperform Humans in Detecting User Zone-Outs? Implications for Intelligent Interfaces
AU - Bosch, Nigel
AU - D'Mello, Sidney K.
N1 - Funding Information:
This article is most closely related to our previous work on automatic face-based detection of mind wandering [11]. We utilized the predictions made by our previously-developed automatic method for the current article, as noted in the article. However, the RQs we propose and address in this article are new. The authors would like to thank Cathlyn Stone for implementing natural language processing code used in this work. This research was supported by the National Science Foundation (NSF) (DRL 1235958, IIS 1523091, DRL 1920510). Any opinions, findings and conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of NSF. Authors’ addresses: N. Bosch, School of Information Sciences and Department of Educational Psychology, University of Illinois at Urbana-Champaign, Champaign, IL, 61820; email: pnb@illinois.edu; S. K. D’Mello, Department of Computer Science and Institute of Cognitive Science, University of Colorado Boulder, Boulder, CO, 80309; email: sidney.dmello@colorado.edu. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2022 Association for Computing Machinery. 1073-0516/2022/01-ART10 $15.00 https://doi.org/10.1145/3481889
Publisher Copyright:
© 2022 Association for Computing Machinery.
PY - 2022/4
Y1 - 2022/4
N2 - The ability to identify whether a user is "zoning out"(mind wandering) from video has many HCI (e.g., distance learning, high-stakes vigilance tasks). However, it remains unknown how well humans can perform this task, how they compare to automatic computerized approaches, and how a fusion of the two might improve accuracy. We analyzed videos of users' faces and upper bodies recorded 10s prior to self-reported mind wandering (i.e., ground truth) while they engaged in a computerized reading task. We found that a state-of-the-art machine learning model had comparable accuracy to aggregated judgments of nine untrained human observers (area under receiver operating characteristic curve [AUC] = .598 versus .589). A fusion of the two (AUC = .644) outperformed each, presumably because each focused on complementary cues. Furthermore, adding more humans beyond 3-4 observers yielded diminishing returns. We discuss implications of human-computer fusion as a means to improve accuracy in complex tasks.
AB - The ability to identify whether a user is "zoning out"(mind wandering) from video has many HCI (e.g., distance learning, high-stakes vigilance tasks). However, it remains unknown how well humans can perform this task, how they compare to automatic computerized approaches, and how a fusion of the two might improve accuracy. We analyzed videos of users' faces and upper bodies recorded 10s prior to self-reported mind wandering (i.e., ground truth) while they engaged in a computerized reading task. We found that a state-of-the-art machine learning model had comparable accuracy to aggregated judgments of nine untrained human observers (area under receiver operating characteristic curve [AUC] = .598 versus .589). A fusion of the two (AUC = .644) outperformed each, presumably because each focused on complementary cues. Furthermore, adding more humans beyond 3-4 observers yielded diminishing returns. We discuss implications of human-computer fusion as a means to improve accuracy in complex tasks.
KW - Mind wandering
KW - attention-aware interfaces
KW - facial expression recognition
KW - human-machine comparison
UR - http://www.scopus.com/inward/record.url?scp=85130418594&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85130418594&partnerID=8YFLogxK
U2 - 10.1145/3481889
DO - 10.1145/3481889
M3 - Article
AN - SCOPUS:85130418594
SN - 1073-0516
VL - 29
JO - ACM Transactions on Computer-Human Interaction
JF - ACM Transactions on Computer-Human Interaction
IS - 2
M1 - 10
ER -