Multimodal approaches for emotion recognition: A survey

Nicu Sebe, Ira Cohen, Theo Gevers, Thomas S Huang

Research output: Contribution to journalConference articlepeer-review


Recent technological advances have enabled human users to interact with computers in ways previously unimaginable. Beyond the confines of the keyboard and mouse, new modalities for human-computer interaction such as voice, gesture, and force-feedback are emerging. Despite important advances, one necessary ingredient for natural interaction is still missing-emotions. Emotions play an important role in human-to-human communication and interaction, allowing people to express themselves beyond the verbal domain. The ability to understand human emotions is desirable for the computer in several applications. This paper explores new ways of human-computer interaction that enable the computer to be more aware of the user's emotional and attentional expressions. We present the basic research in the field and the recent advances into the emotion recognition from facial, voice, and pshysiological signals, where the different modalities are treated independently. We then describe the challenging problem of multimodal emotion recognition and we advocate the use of probabilistic graphical models when fusing the different modalities. We also discuss the difficult issues of obtaining reliable affective data, obtaining ground truth for emotion recognition, and the use of unlabeled data.

Original languageEnglish (US)
Article number08
Pages (from-to)56-67
Number of pages12
JournalProceedings of SPIE - The International Society for Optical Engineering
StatePublished - Jul 20 2005
EventProceedings of SPIE-IS and T Electronic Imaging - Internet Imaging VI - San Jose, CA, United States
Duration: Jan 18 2005Jan 20 2005


  • Emotion recognition
  • Human-computer interaction
  • Multimodal approach

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering


Dive into the research topics of 'Multimodal approaches for emotion recognition: A survey'. Together they form a unique fingerprint.

Cite this