Gaze correction for 3D tele-immersive communication system

Wei Yong Eng, Dongbo Min, Viet Anh Nguyen, Jiangbo Lu, Minh N. Do

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The lack of eye contact between participants in a tele-conferencing makes nonverbal communication unnatural and ineffective. A lot of research has focused on correcting the user gaze for a natural communication. Most of prior solutions require expensive and bulky hardware, or incorporate a complicated algorithm causing inefficiency and deployment. In this paper, we propose an effective and efficient gaze correction solution for a 3D tele-conferencing system in a single color/depth camera set-up. A raw depth map is first refined using the corresponding color image. Then, both color and depth data of the participant are accurately segmented. A novel view is synthesized in the location of the display screen which coincides with the user gaze. Stereoscopic views, i.e. virtual left and right images, can also be generated for 3D immersive conferencing, and are displayed in a 3D monitor with 3D virtual background scenes. Finally, to handle large hole regions that often occur in the view synthesized with a single color camera, we propose a simple yet robust hole filling technique that works in real-time. This novel inpainting method can effectively reconstruct missing parts of the synthesized image under various challenging situations. Our proposed system works in real-time on a single core CPU without requiring dedicated hardware, including data acquisition, post-processing, rendering, and so on.

Original languageEnglish (US)
Title of host publication2013 IEEE 11th IVMSP Workshop
Subtitle of host publication3D Image/Video Technologies and Applications, IVMSP 2013 - Proceedings
DOIs
StatePublished - Nov 28 2013
Event2013 IEEE 11th Workshop on 3D Image/Video Technologies and Applications, IVMSP 2013 - Seoul, Korea, Republic of
Duration: Jun 10 2013Jun 12 2013

Publication series

Name2013 IEEE 11th IVMSP Workshop: 3D Image/Video Technologies and Applications, IVMSP 2013 - Proceedings

Other

Other2013 IEEE 11th Workshop on 3D Image/Video Technologies and Applications, IVMSP 2013
CountryKorea, Republic of
CitySeoul
Period6/10/136/12/13

Keywords

  • depth camera
  • depth image based rendering (DIBR)
  • foreground segmentation
  • Gaze correction
  • tele-conferencing

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Computer Vision and Pattern Recognition
  • Computer Science Applications

Fingerprint Dive into the research topics of 'Gaze correction for 3D tele-immersive communication system'. Together they form a unique fingerprint.

  • Cite this

    Eng, W. Y., Min, D., Nguyen, V. A., Lu, J., & Do, M. N. (2013). Gaze correction for 3D tele-immersive communication system. In 2013 IEEE 11th IVMSP Workshop: 3D Image/Video Technologies and Applications, IVMSP 2013 - Proceedings [6611942] (2013 IEEE 11th IVMSP Workshop: 3D Image/Video Technologies and Applications, IVMSP 2013 - Proceedings). https://doi.org/10.1109/IVMSPW.2013.6611942