It's written on your face: Detecting affective states from facial expressions while learning computer programming

Nigel Bosch; Yuxuan Chen; Sidney D'Mello

doi:10.1007/978-3-319-07221-0_5

It's written on your face: Detecting affective states from facial expressions while learning computer programming

Nigel Bosch, Yuxuan Chen, Sidney D'Mello

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We built detectors capable of automatically recognizing affective states of novice computer programmers from student-annotated videos of their faces recorded during an introductory programming tutoring session. We used the Computer Expression Recognition Toolbox (CERT) to track facial features based on the Facial Action Coding System, and machine learning techniques to build classification models. Confusion/Uncertainty and Frustration were distinguished from all other affective states in a student-independent fashion at levels above chance (Cohen's kappa =.22 and.23, respectively), but detection accuracies for Boredom, Flow/Engagement, and Neutral were lower (kappas =.04,.11, and.07). We discuss the differences between detection of spontaneous versus fixed (polled) judgments as well as the features used in the models.

Original language	English (US)
Title of host publication	Intelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings
Publisher	Springer-Verlag
Pages	39-44
Number of pages	6
ISBN (Print)	9783319072203
DOIs	https://doi.org/10.1007/978-3-319-07221-0_5
State	Published - Jan 1 2014
Externally published	Yes
Event	12th International Conference on Intelligent Tutoring Systems, ITS 2014 - Honolulu, HI, United States Duration: Jun 5 2014 → Jun 9 2014

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	8474 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Other

Other	12th International Conference on Intelligent Tutoring Systems, ITS 2014
Country	United States
City	Honolulu, HI
Period	6/5/14 → 6/9/14

Fingerprint

ASJC Scopus subject areas

Theoretical Computer Science
Computer Science(all)

Cite this

Bosch, N., Chen, Y., & D'Mello, S. (2014). It's written on your face: Detecting affective states from facial expressions while learning computer programming. In Intelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings (pp. 39-44). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8474 LNCS). Springer-Verlag. https://doi.org/10.1007/978-3-319-07221-0_5

It's written on your face : Detecting affective states from facial expressions while learning computer programming. / Bosch, Nigel; Chen, Yuxuan; D'Mello, Sidney.

Intelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings. Springer-Verlag, 2014. p. 39-44 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8474 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Bosch, N, Chen, Y & D'Mello, S 2014, It's written on your face: Detecting affective states from facial expressions while learning computer programming. in Intelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8474 LNCS, Springer-Verlag, pp. 39-44, 12th International Conference on Intelligent Tutoring Systems, ITS 2014, Honolulu, HI, United States, 6/5/14. https://doi.org/10.1007/978-3-319-07221-0_5

Bosch N, Chen Y, D'Mello S. It's written on your face: Detecting affective states from facial expressions while learning computer programming. In Intelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings. Springer-Verlag. 2014. p. 39-44. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-07221-0_5

Bosch, Nigel ; Chen, Yuxuan ; D'Mello, Sidney. / It's written on your face : Detecting affective states from facial expressions while learning computer programming. Intelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings. Springer-Verlag, 2014. pp. 39-44 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{c5edb9b2774647f888d59f5938129c2e,

title = "It's written on your face: Detecting affective states from facial expressions while learning computer programming",

abstract = "We built detectors capable of automatically recognizing affective states of novice computer programmers from student-annotated videos of their faces recorded during an introductory programming tutoring session. We used the Computer Expression Recognition Toolbox (CERT) to track facial features based on the Facial Action Coding System, and machine learning techniques to build classification models. Confusion/Uncertainty and Frustration were distinguished from all other affective states in a student-independent fashion at levels above chance (Cohen's kappa =.22 and.23, respectively), but detection accuracies for Boredom, Flow/Engagement, and Neutral were lower (kappas =.04,.11, and.07). We discuss the differences between detection of spontaneous versus fixed (polled) judgments as well as the features used in the models.",

author = "Nigel Bosch and Yuxuan Chen and Sidney D'Mello",

year = "2014",

month = jan

day = "1",

doi = "10.1007/978-3-319-07221-0_5",

language = "English (US)",

isbn = "9783319072203",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer-Verlag",

pages = "39--44",

booktitle = "Intelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings",

note = "12th International Conference on Intelligent Tutoring Systems, ITS 2014 ; Conference date: 05-06-2014 Through 09-06-2014",

}

TY - GEN

T1 - It's written on your face

T2 - 12th International Conference on Intelligent Tutoring Systems, ITS 2014

AU - Bosch, Nigel

AU - Chen, Yuxuan

AU - D'Mello, Sidney

PY - 2014/1/1

Y1 - 2014/1/1

N2 - We built detectors capable of automatically recognizing affective states of novice computer programmers from student-annotated videos of their faces recorded during an introductory programming tutoring session. We used the Computer Expression Recognition Toolbox (CERT) to track facial features based on the Facial Action Coding System, and machine learning techniques to build classification models. Confusion/Uncertainty and Frustration were distinguished from all other affective states in a student-independent fashion at levels above chance (Cohen's kappa =.22 and.23, respectively), but detection accuracies for Boredom, Flow/Engagement, and Neutral were lower (kappas =.04,.11, and.07). We discuss the differences between detection of spontaneous versus fixed (polled) judgments as well as the features used in the models.

AB - We built detectors capable of automatically recognizing affective states of novice computer programmers from student-annotated videos of their faces recorded during an introductory programming tutoring session. We used the Computer Expression Recognition Toolbox (CERT) to track facial features based on the Facial Action Coding System, and machine learning techniques to build classification models. Confusion/Uncertainty and Frustration were distinguished from all other affective states in a student-independent fashion at levels above chance (Cohen's kappa =.22 and.23, respectively), but detection accuracies for Boredom, Flow/Engagement, and Neutral were lower (kappas =.04,.11, and.07). We discuss the differences between detection of spontaneous versus fixed (polled) judgments as well as the features used in the models.

UR - http://www.scopus.com/inward/record.url?scp=84958530004&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84958530004&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-07221-0_5

DO - 10.1007/978-3-319-07221-0_5

M3 - Conference contribution

AN - SCOPUS:84958530004

SN - 9783319072203

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 39

EP - 44

BT - Intelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings

PB - Springer-Verlag

Y2 - 5 June 2014 through 9 June 2014

ER -

Access to Document

10.1007/978-3-319-07221-0_5