Bimodal HCI-related affect recognition

Zhihong Zeng, Juin Tu, Ming Liu, Tong Zhang, Nicholas Rizzolo, Zhenqiu Zhang, Thomas S. Huang, Dan Roth, Stephen Levinson

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Perhaps the most fundamental application of affective computing would be Human-Computer Interaction (HCI) in which the computer is able to detect and track the user's affective states, and make corresponding feedback. The human multi-sensor affect system defines the expectation of multimodal affect analyzer. In this paper, we present our efforts toward audio-visual HCI-related affect recognition. With HCI applications in mind, we take into account some special affective states which indicate users' cognitive/motivational states. Facing the fact that a facial expression is influenced by both an affective state and speech content, we apply a smoothing method to extract the information of the affective state from facial features. In our fusion stage, a voting method is applied to combine audio and visual modalities so that the final affect recognition accuracy is greatly improved. We test our bimodal affect recognition approach on 38 subjects with 11 HCI-related affect states. The extensive experimental results show that the average person-dependent affect recognition accuracy is almost 90% for our bimodal fusion.

Original languageEnglish (US)
Title of host publicationICMI'04 - Sixth International Conference on Multimodal Interfaces
Pages137-143
Number of pages7
StatePublished - Dec 1 2004
EventICMI'04 - Sixth International Conference on Multimodal Interfaces - , United States
Duration: Oct 14 2004Oct 15 2004

Publication series

NameICMI'04 - Sixth International Conference on Multimodal Interfaces

Other

OtherICMI'04 - Sixth International Conference on Multimodal Interfaces
CountryUnited States
Period10/14/0410/15/04

Fingerprint

Human computer interaction
Fusion reactions
Sensor data fusion
Feedback

Keywords

  • Affect recognition
  • Affective computing
  • Emotion recognition
  • HCI
  • Multimodal human-computer interaction

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Zeng, Z., Tu, J., Liu, M., Zhang, T., Rizzolo, N., Zhang, Z., ... Levinson, S. (2004). Bimodal HCI-related affect recognition. In ICMI'04 - Sixth International Conference on Multimodal Interfaces (pp. 137-143). (ICMI'04 - Sixth International Conference on Multimodal Interfaces).

Bimodal HCI-related affect recognition. / Zeng, Zhihong; Tu, Juin; Liu, Ming; Zhang, Tong; Rizzolo, Nicholas; Zhang, Zhenqiu; Huang, Thomas S.; Roth, Dan; Levinson, Stephen.

ICMI'04 - Sixth International Conference on Multimodal Interfaces. 2004. p. 137-143 (ICMI'04 - Sixth International Conference on Multimodal Interfaces).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Zeng, Z, Tu, J, Liu, M, Zhang, T, Rizzolo, N, Zhang, Z, Huang, TS, Roth, D & Levinson, S 2004, Bimodal HCI-related affect recognition. in ICMI'04 - Sixth International Conference on Multimodal Interfaces. ICMI'04 - Sixth International Conference on Multimodal Interfaces, pp. 137-143, ICMI'04 - Sixth International Conference on Multimodal Interfaces, United States, 10/14/04.
Zeng Z, Tu J, Liu M, Zhang T, Rizzolo N, Zhang Z et al. Bimodal HCI-related affect recognition. In ICMI'04 - Sixth International Conference on Multimodal Interfaces. 2004. p. 137-143. (ICMI'04 - Sixth International Conference on Multimodal Interfaces).
Zeng, Zhihong ; Tu, Juin ; Liu, Ming ; Zhang, Tong ; Rizzolo, Nicholas ; Zhang, Zhenqiu ; Huang, Thomas S. ; Roth, Dan ; Levinson, Stephen. / Bimodal HCI-related affect recognition. ICMI'04 - Sixth International Conference on Multimodal Interfaces. 2004. pp. 137-143 (ICMI'04 - Sixth International Conference on Multimodal Interfaces).
@inproceedings{c7d07b1a46e54abfb9014546d53fd0dd,
title = "Bimodal HCI-related affect recognition",
abstract = "Perhaps the most fundamental application of affective computing would be Human-Computer Interaction (HCI) in which the computer is able to detect and track the user's affective states, and make corresponding feedback. The human multi-sensor affect system defines the expectation of multimodal affect analyzer. In this paper, we present our efforts toward audio-visual HCI-related affect recognition. With HCI applications in mind, we take into account some special affective states which indicate users' cognitive/motivational states. Facing the fact that a facial expression is influenced by both an affective state and speech content, we apply a smoothing method to extract the information of the affective state from facial features. In our fusion stage, a voting method is applied to combine audio and visual modalities so that the final affect recognition accuracy is greatly improved. We test our bimodal affect recognition approach on 38 subjects with 11 HCI-related affect states. The extensive experimental results show that the average person-dependent affect recognition accuracy is almost 90{\%} for our bimodal fusion.",
keywords = "Affect recognition, Affective computing, Emotion recognition, HCI, Multimodal human-computer interaction",
author = "Zhihong Zeng and Juin Tu and Ming Liu and Tong Zhang and Nicholas Rizzolo and Zhenqiu Zhang and Huang, {Thomas S.} and Dan Roth and Stephen Levinson",
year = "2004",
month = "12",
day = "1",
language = "English (US)",
isbn = "1581139543",
series = "ICMI'04 - Sixth International Conference on Multimodal Interfaces",
pages = "137--143",
booktitle = "ICMI'04 - Sixth International Conference on Multimodal Interfaces",

}

TY - GEN

T1 - Bimodal HCI-related affect recognition

AU - Zeng, Zhihong

AU - Tu, Juin

AU - Liu, Ming

AU - Zhang, Tong

AU - Rizzolo, Nicholas

AU - Zhang, Zhenqiu

AU - Huang, Thomas S.

AU - Roth, Dan

AU - Levinson, Stephen

PY - 2004/12/1

Y1 - 2004/12/1

N2 - Perhaps the most fundamental application of affective computing would be Human-Computer Interaction (HCI) in which the computer is able to detect and track the user's affective states, and make corresponding feedback. The human multi-sensor affect system defines the expectation of multimodal affect analyzer. In this paper, we present our efforts toward audio-visual HCI-related affect recognition. With HCI applications in mind, we take into account some special affective states which indicate users' cognitive/motivational states. Facing the fact that a facial expression is influenced by both an affective state and speech content, we apply a smoothing method to extract the information of the affective state from facial features. In our fusion stage, a voting method is applied to combine audio and visual modalities so that the final affect recognition accuracy is greatly improved. We test our bimodal affect recognition approach on 38 subjects with 11 HCI-related affect states. The extensive experimental results show that the average person-dependent affect recognition accuracy is almost 90% for our bimodal fusion.

AB - Perhaps the most fundamental application of affective computing would be Human-Computer Interaction (HCI) in which the computer is able to detect and track the user's affective states, and make corresponding feedback. The human multi-sensor affect system defines the expectation of multimodal affect analyzer. In this paper, we present our efforts toward audio-visual HCI-related affect recognition. With HCI applications in mind, we take into account some special affective states which indicate users' cognitive/motivational states. Facing the fact that a facial expression is influenced by both an affective state and speech content, we apply a smoothing method to extract the information of the affective state from facial features. In our fusion stage, a voting method is applied to combine audio and visual modalities so that the final affect recognition accuracy is greatly improved. We test our bimodal affect recognition approach on 38 subjects with 11 HCI-related affect states. The extensive experimental results show that the average person-dependent affect recognition accuracy is almost 90% for our bimodal fusion.

KW - Affect recognition

KW - Affective computing

KW - Emotion recognition

KW - HCI

KW - Multimodal human-computer interaction

UR - http://www.scopus.com/inward/record.url?scp=14944338502&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=14944338502&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:14944338502

SN - 1581139543

T3 - ICMI'04 - Sixth International Conference on Multimodal Interfaces

SP - 137

EP - 143

BT - ICMI'04 - Sixth International Conference on Multimodal Interfaces

ER -