It's written on your face: Detecting affective states from facial expressions while learning computer programming

Nigel Bosch, Yuxuan Chen, Sidney D'Mello

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We built detectors capable of automatically recognizing affective states of novice computer programmers from student-annotated videos of their faces recorded during an introductory programming tutoring session. We used the Computer Expression Recognition Toolbox (CERT) to track facial features based on the Facial Action Coding System, and machine learning techniques to build classification models. Confusion/Uncertainty and Frustration were distinguished from all other affective states in a student-independent fashion at levels above chance (Cohen's kappa =.22 and.23, respectively), but detection accuracies for Boredom, Flow/Engagement, and Neutral were lower (kappas =.04,.11, and.07). We discuss the differences between detection of spontaneous versus fixed (polled) judgments as well as the features used in the models.

Original languageEnglish (US)
Title of host publicationIntelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings
PublisherSpringer-Verlag
Pages39-44
Number of pages6
ISBN (Print)9783319072203
DOIs
StatePublished - Jan 1 2014
Externally publishedYes
Event12th International Conference on Intelligent Tutoring Systems, ITS 2014 - Honolulu, HI, United States
Duration: Jun 5 2014Jun 9 2014

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8474 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other12th International Conference on Intelligent Tutoring Systems, ITS 2014
CountryUnited States
CityHonolulu, HI
Period6/5/146/9/14

    Fingerprint

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Bosch, N., Chen, Y., & D'Mello, S. (2014). It's written on your face: Detecting affective states from facial expressions while learning computer programming. In Intelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings (pp. 39-44). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8474 LNCS). Springer-Verlag. https://doi.org/10.1007/978-3-319-07221-0_5