Bimodal HCI-related affect recognition

Zhihong Zeng, Juin Tu, Ming Liu, Tong Zhang, Nicholas Rizzolo, Zhenqiu Zhang, Thomas S. Huang, Dan Roth, Stephen Levinson

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Perhaps the most fundamental application of affective computing would be Human-Computer Interaction (HCI) in which the computer is able to detect and track the user's affective states, and make corresponding feedback. The human multi-sensor affect system defines the expectation of multimodal affect analyzer. In this paper, we present our efforts toward audio-visual HCI-related affect recognition. With HCI applications in mind, we take into account some special affective states which indicate users' cognitive/motivational states. Facing the fact that a facial expression is influenced by both an affective state and speech content, we apply a smoothing method to extract the information of the affective state from facial features. In our fusion stage, a voting method is applied to combine audio and visual modalities so that the final affect recognition accuracy is greatly improved. We test our bimodal affect recognition approach on 38 subjects with 11 HCI-related affect states. The extensive experimental results show that the average person-dependent affect recognition accuracy is almost 90% for our bimodal fusion.

Original languageEnglish (US)
Title of host publicationICMI'04 - Sixth International Conference on Multimodal Interfaces
PublisherAssociation for Computing Machinery
Number of pages7
ISBN (Print)1581139543, 9781581139549
StatePublished - 2004
Externally publishedYes
EventICMI'04 - Sixth International Conference on Multimodal Interfaces - , United States
Duration: Oct 14 2004Oct 15 2004

Publication series

NameICMI'04 - Sixth International Conference on Multimodal Interfaces


OtherICMI'04 - Sixth International Conference on Multimodal Interfaces
Country/TerritoryUnited States


  • Affect recognition
  • Affective computing
  • Emotion recognition
  • HCI
  • Multimodal human-computer interaction

ASJC Scopus subject areas

  • Engineering(all)


Dive into the research topics of 'Bimodal HCI-related affect recognition'. Together they form a unique fingerprint.

Cite this