Real-time speech-driven face animation with expressions using neural networks

Pengyu Hong, Zhen Wen, Thomas S. Huang

Research output: Contribution to journalArticlepeer-review


A real-time speech-driven synthetic talking face provides an effective multimodal communication interface in distributed collaboration environments. Nonverbal gestures such as facial expressions are important to human communication and should be considered by speech-driven face animation systems. In this paper, we present a framework that systematically addresses facial deformation modeling, automatic facial motion analysis, and real-time speech-driven face animation with expression using neural networks. Based on this framework, we learn a quantitative visual representation of the facial deformations, called the motion units (MUs). An facial deformation can be approximated by a linear combination of the MUs weighted by MU parameters (MUPs). We develop an MU-based facial motion tracking algorithm which is used to collect an audio-visual training database. Then, we construct a real-time audio-to-MUP mapping by training a set of neural networks using the collected audio-visual training database. The quantitative evaluation of the mapping shows the effectiveness of the proposed approach. Using the proposed method, we develop the functionality of real-time speech-driven face animation with expressions for the iFACE system. Experimental results show that the synthetic expressive talking face of the iFACE system is comparable with a real face in terms of the effectiveness of their influences on bimodal human emotion perception.

Original languageEnglish (US)
Pages (from-to)916-927
Number of pages12
JournalIEEE Transactions on Neural Networks
Issue number4
StatePublished - Jul 2002


  • Facial deformation modeling
  • Facial motion analysis and synthesis
  • Neural networks
  • Real-time speech-driven talking face with expressions

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence


Dive into the research topics of 'Real-time speech-driven face animation with expressions using neural networks'. Together they form a unique fingerprint.

Cite this