ITEM: Immersive telepresence for entertainment and meetings - A practical approach

Viet Anh Nguyen, Jiangbo Lu, Shengkui Zhao, Dung T. Vu, Hongsheng Yang, Douglas L. Jones, Minh N. Do

Research output: Contribution to journalArticle

Abstract

This paper presents an immersive telepresence system for entertainment and meetings (ITEM). The system aims to provide a radically new video communication experience by seamlessly merging participants into the same virtual space to allow a natural interaction among them and shared collaborative contents. With the goal to make a scalable, flexible system for various business solutions as well as easily accessible by massive consumers, we address the challenges in the whole pipeline of media processing, communication, and displaying in our design and realization of such a system. Particularly, in this paper we focus on the system aspects that maximize the end-user experience, optimize the system and network resources, and enable various teleimmersive (TI) application scenarios. In addition, we also present a few key technologies, i.e., fast object-based video coding for real world data and spatialized audio capture and 3-D sound localization for group teleconferencing. Our effort is to investigate and optimize the key system components and provide an efficient end-to-end optimization and integration by considering user needs and preferences. Extensive experiments show the developed system runs reliably and comfortably in real time with a minimal setup requirement (e.g., a webcam or a color plus depth camera, an optional microphone array, a laptop/desktop connected to the Internet) for TI communication. With such a really minimal deployment requirement, we present a variety of interesting applications and user experiences created by ITEM.

Original languageEnglish (US)
Article number6971053
Pages (from-to)546-561
Number of pages16
JournalIEEE Journal on Selected Topics in Signal Processing
Volume9
Issue number3
DOIs
StatePublished - Apr 1 2015

Fingerprint

Communication
Teleconferencing
Microphones
Image coding
Merging
Pipelines
Cameras
Acoustic waves
Internet
Color
Processing
Industry
Experiments

Keywords

  • 3D sound localization
  • 3D spatialized audio
  • Object-based coding
  • Teleimmersive system
  • Video
  • Video conferencing
  • object cutout

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering

Cite this

ITEM : Immersive telepresence for entertainment and meetings - A practical approach. / Nguyen, Viet Anh; Lu, Jiangbo; Zhao, Shengkui; Vu, Dung T.; Yang, Hongsheng; Jones, Douglas L.; Do, Minh N.

In: IEEE Journal on Selected Topics in Signal Processing, Vol. 9, No. 3, 6971053, 01.04.2015, p. 546-561.

Research output: Contribution to journalArticle

Nguyen, Viet Anh ; Lu, Jiangbo ; Zhao, Shengkui ; Vu, Dung T. ; Yang, Hongsheng ; Jones, Douglas L. ; Do, Minh N. / ITEM : Immersive telepresence for entertainment and meetings - A practical approach. In: IEEE Journal on Selected Topics in Signal Processing. 2015 ; Vol. 9, No. 3. pp. 546-561.
@article{e29e460a7b664ca7a6997c8ea233c448,
title = "ITEM: Immersive telepresence for entertainment and meetings - A practical approach",
abstract = "This paper presents an immersive telepresence system for entertainment and meetings (ITEM). The system aims to provide a radically new video communication experience by seamlessly merging participants into the same virtual space to allow a natural interaction among them and shared collaborative contents. With the goal to make a scalable, flexible system for various business solutions as well as easily accessible by massive consumers, we address the challenges in the whole pipeline of media processing, communication, and displaying in our design and realization of such a system. Particularly, in this paper we focus on the system aspects that maximize the end-user experience, optimize the system and network resources, and enable various teleimmersive (TI) application scenarios. In addition, we also present a few key technologies, i.e., fast object-based video coding for real world data and spatialized audio capture and 3-D sound localization for group teleconferencing. Our effort is to investigate and optimize the key system components and provide an efficient end-to-end optimization and integration by considering user needs and preferences. Extensive experiments show the developed system runs reliably and comfortably in real time with a minimal setup requirement (e.g., a webcam or a color plus depth camera, an optional microphone array, a laptop/desktop connected to the Internet) for TI communication. With such a really minimal deployment requirement, we present a variety of interesting applications and user experiences created by ITEM.",
keywords = "3D sound localization, 3D spatialized audio, Object-based coding, Teleimmersive system, Video, Video conferencing, object cutout",
author = "Nguyen, {Viet Anh} and Jiangbo Lu and Shengkui Zhao and Vu, {Dung T.} and Hongsheng Yang and Jones, {Douglas L.} and Do, {Minh N.}",
year = "2015",
month = "4",
day = "1",
doi = "10.1109/JSTSP.2014.2375819",
language = "English (US)",
volume = "9",
pages = "546--561",
journal = "IEEE Journal on Selected Topics in Signal Processing",
issn = "1932-4553",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "3",

}

TY - JOUR

T1 - ITEM

T2 - Immersive telepresence for entertainment and meetings - A practical approach

AU - Nguyen, Viet Anh

AU - Lu, Jiangbo

AU - Zhao, Shengkui

AU - Vu, Dung T.

AU - Yang, Hongsheng

AU - Jones, Douglas L.

AU - Do, Minh N.

PY - 2015/4/1

Y1 - 2015/4/1

N2 - This paper presents an immersive telepresence system for entertainment and meetings (ITEM). The system aims to provide a radically new video communication experience by seamlessly merging participants into the same virtual space to allow a natural interaction among them and shared collaborative contents. With the goal to make a scalable, flexible system for various business solutions as well as easily accessible by massive consumers, we address the challenges in the whole pipeline of media processing, communication, and displaying in our design and realization of such a system. Particularly, in this paper we focus on the system aspects that maximize the end-user experience, optimize the system and network resources, and enable various teleimmersive (TI) application scenarios. In addition, we also present a few key technologies, i.e., fast object-based video coding for real world data and spatialized audio capture and 3-D sound localization for group teleconferencing. Our effort is to investigate and optimize the key system components and provide an efficient end-to-end optimization and integration by considering user needs and preferences. Extensive experiments show the developed system runs reliably and comfortably in real time with a minimal setup requirement (e.g., a webcam or a color plus depth camera, an optional microphone array, a laptop/desktop connected to the Internet) for TI communication. With such a really minimal deployment requirement, we present a variety of interesting applications and user experiences created by ITEM.

AB - This paper presents an immersive telepresence system for entertainment and meetings (ITEM). The system aims to provide a radically new video communication experience by seamlessly merging participants into the same virtual space to allow a natural interaction among them and shared collaborative contents. With the goal to make a scalable, flexible system for various business solutions as well as easily accessible by massive consumers, we address the challenges in the whole pipeline of media processing, communication, and displaying in our design and realization of such a system. Particularly, in this paper we focus on the system aspects that maximize the end-user experience, optimize the system and network resources, and enable various teleimmersive (TI) application scenarios. In addition, we also present a few key technologies, i.e., fast object-based video coding for real world data and spatialized audio capture and 3-D sound localization for group teleconferencing. Our effort is to investigate and optimize the key system components and provide an efficient end-to-end optimization and integration by considering user needs and preferences. Extensive experiments show the developed system runs reliably and comfortably in real time with a minimal setup requirement (e.g., a webcam or a color plus depth camera, an optional microphone array, a laptop/desktop connected to the Internet) for TI communication. With such a really minimal deployment requirement, we present a variety of interesting applications and user experiences created by ITEM.

KW - 3D sound localization

KW - 3D spatialized audio

KW - Object-based coding

KW - Teleimmersive system

KW - Video

KW - Video conferencing

KW - object cutout

UR - http://www.scopus.com/inward/record.url?scp=84938912008&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84938912008&partnerID=8YFLogxK

U2 - 10.1109/JSTSP.2014.2375819

DO - 10.1109/JSTSP.2014.2375819

M3 - Article

AN - SCOPUS:84938912008

VL - 9

SP - 546

EP - 561

JO - IEEE Journal on Selected Topics in Signal Processing

JF - IEEE Journal on Selected Topics in Signal Processing

SN - 1932-4553

IS - 3

M1 - 6971053

ER -