Automatic Detection of Mind Wandering from Video in the Lab and in the Classroom

Nigel Bosch, Sidney K. D'Mello

Research output: Contribution to journalArticlepeer-review


We report two studies that used facial features to automatically detect mind wandering, a ubiquitous phenomenon whereby attention drifts from the current task to unrelated thoughts. In a laboratory study, university students $(N = 152)$(N=152) read a scientific text, whereas in a classroom study high school students $(N = 135)$(N=135) learned biology from an intelligent tutoring system. Mind wandering was measured using validated self-report methods. In the lab, we recorded face videos and analyzed these at six levels of granularity: (1) upper-body movement; (2) head pose; (3) facial textures; (4) facial action units (AUs); (5) co-occurring AUs; and (6) temporal dynamics of AUs. Due to privacy constraints, videos were not recorded in the classroom. Instead, we extracted head pose, AUs, and AU co-occurrences in real-time. Machine learning models, consisting of support vector machines (SVM) and deep neural networks, achieved $F_{1}$F1 scores of.478 and.414 (25.4 and 20.9 percent above-chance improvements, both with SVMs) for detecting mind wandering in the lab and classroom, respectively. The lab-based detectors achieved 8.4 percent improvement over the previous state-of-the-art; no comparison is available for classroom detectors. We discuss how the detectors can integrate into intelligent interfaces to increase engagement and learning by responding to wandering minds.

Original languageEnglish (US)
Pages (from-to)974-988
Number of pages15
JournalIEEE Transactions on Affective Computing
Issue number4
StatePublished - 2021


  • Affective computing
  • computer vision
  • educational technology
  • human-computer interaction

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction


Dive into the research topics of 'Automatic Detection of Mind Wandering from Video in the Lab and in the Classroom'. Together they form a unique fingerprint.

Cite this