Improving affect detection in game-based learning with multimodal data fusion

Nathan Henderson, Jonathan Rowe, Luc Paquette, Ryan S. Baker, James Lester

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Accurately recognizing learner affect is critically important for enabling affect-responsive learning environments to support student learning and engagement. Multimodal affect detection combining sensor-based and sensor-free approaches has shown significant promise in both laboratory and classroom settings. However, important questions remain regarding which data channels are most predictive and how they should be combined. In this paper, we investigate a multimodal affect detection framework that integrates motion tracking-based posture data and interaction-based trace data to recognize the affective states of students engaged with a game-based learning environment for emergency medical training. We compare several machine learning-based affective models using competing feature-level and decision-level multimodal data fusion approaches. Results indicate that multimodal affect detectors induced using joint feature representations from posture-based and interaction-based data channels yield improved accuracy relative to unimodal models across several learner-centered affective states. These findings point toward implications for the design of multimodal affect-responsive learning environments that support learning and engagement.

Original languageEnglish (US)
Title of host publicationArtificial Intelligence in Education- 21st International Conference, AIED 2020, Proceedings, Part I
EditorsIg Ibert Bittencourt, Mutlu Cukurova, Rose Luckin, Kasia Muldner, Eva Millán
Number of pages12
ISBN (Print)9783030522360
StatePublished - 2020
Event21st International Conference on Artificial Intelligence in Education, AIED 2020 - Ifrane, Morocco
Duration: Jul 6 2020Jul 10 2020

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12163 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference21st International Conference on Artificial Intelligence in Education, AIED 2020


  • Affect detection
  • Game-based learning
  • Multimodal data fusion

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)


Dive into the research topics of 'Improving affect detection in game-based learning with multimodal data fusion'. Together they form a unique fingerprint.

Cite this