Real-time multimodal humanavatar interaction

Yun Fu, Renxiang Li, Thomas S Huang, Mike Danielsen

Research output: Contribution to journalArticle


This paper presents a novel real-time multimodal human-avatar interaction (RTM-HAI) framework with vision-based remote animation control (RAC). The framework is designed for both mobile and desktop avatar-based human-machine or human-human visual communications in real-world scenarios. Using 3-D components stored in the Java mobile 3-D (M3G) file format, the avatar models can be flexibly constructed and customized on the fly on any mobile devices or systems that support the M3G standard. For the RAC head tracker, we propose a 2-D real-time face detection/tracking strategy through an interactive loop, in which the detection and tracking complement each other for efficient and reliable face localization, tolerating extreme user movement. With the face location robustly tracked, the RAC head tracker selects a main user and estimates the user's head rolling, tilting, yawing, scaling, horizontal, and vertical motion in order to generate avatar animation parameters. The animation parameters can be used either locally or remotely and can be transmitted through socket over the network. In addition, it integrates audio-visual analysis and synthesis modules to realize multichannel and runtime animations, visual TTS and real-time viseme detection and rendering. The framework is recognized as an effective design for future realistic industrial products of humanoid kiosk and human-to-human mobile communication.

Original languageEnglish (US)
Article number4449082
Pages (from-to)467-477
Number of pages11
JournalIEEE Transactions on Circuits and Systems for Video Technology
Issue number4
StatePublished - Apr 1 2008


  • Avatar
  • DAZ3D
  • DAZ3D,
  • Head tracking
  • Human-computer interaction
  • M3G
  • Mobile 3-D (M3G)
  • Multimodal system
  • Real-time multi modal human-avatar interaction (RTM-HAI)
  • TTS
  • Visual communication

ASJC Scopus subject areas

  • Media Technology
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Real-time multimodal humanavatar interaction'. Together they form a unique fingerprint.

  • Cite this