Accuracy vs. Availability heuristic in multimodal affect detection in the wild

Nigel Bosch, Huili Chen, Ryan Baker, Valerie Shute, Sidney D'mello

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper discusses multimodal affect detection from a fusion of facial expressions and interaction features derived from students' interactions with an educational game in the noisy real-world context of a computer-enabled classroom. Log data of students' interactions with the game and face videos from 133 students were recorded in a computer-enabled classroom over a two day period. Human observers live annotated learning-centered affective states such as engagement, confusion, and frustration. The face-only detectors were more accurate than interaction-only detectors. Multimodal affect detectors did not show any substantial improvement in accuracy over the face-only detectors. However, the face-only detectors were only applicable to 65% of the cases due to face registration errors caused by excessive movement, occlusion, poor lighting, and other factors. Multimodal fusion techniques were able to improve the applicability of detectors to 98% of cases without sacrificing classification accuracy. Balancing the accuracy vs. applicability tradeoff appears to be an important feature of multimodal affect detection.

Original languageEnglish (US)
Title of host publicationICMI 2015 - Proceedings of the 2015 ACM International Conference on Multimodal Interaction
PublisherAssociation for Computing Machinery, Inc
Pages267-274
Number of pages8
ISBN (Electronic)9781450339124
DOIs
StatePublished - Nov 9 2015
Externally publishedYes
EventACM International Conference on Multimodal Interaction, ICMI 2015 - Seattle, United States
Duration: Nov 9 2015Nov 13 2015

Publication series

NameICMI 2015 - Proceedings of the 2015 ACM International Conference on Multimodal Interaction

Conference

ConferenceACM International Conference on Multimodal Interaction, ICMI 2015
CountryUnited States
CitySeattle
Period11/9/1511/13/15

Keywords

  • Affect
  • Affect detection
  • Facial expressions
  • Interaction
  • Missing data

ASJC Scopus subject areas

  • Computer Science Applications
  • Computer Vision and Pattern Recognition
  • Hardware and Architecture
  • Human-Computer Interaction

Fingerprint Dive into the research topics of 'Accuracy vs. Availability heuristic in multimodal affect detection in the wild'. Together they form a unique fingerprint.

Cite this