Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming

Shu Shi, Cheng Hsin Hsu, Klara Nahrstedt, Roy H. Campbell

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The emerging cloud gaming service has been growing rapidly, but not yet able to reach mobile customers due to many limitations, such as bandwidth and latency. We introduce a 3D image warping assisted real-time video coding method that can potentially meet all the requirements of mobile cloud gaming. The proposed video encoder selects a set of key frames in the video sequence, uses the 3D image warping algorithm to interpolate other non-key frames, and encodes the key frames and the residues frames with an H.264/AVC encoder. Our approach is novel in taking advantage of the run-time graphics rendering contexts (rendering viewpoint, pixel depth, camera motion, etc.) from the 3D game engine to enhance the performance of video encoding for the cloud gaming service. The experiments indicate that our proposed video encoder has the potential to beat the state-of-art x264 encoder in the scenario of real-time cloud gaming. For example, by implementing the proposed method in a 3D tank battle game, we experimentally show that more than 2 dB quality improvement is possible.

Original languageEnglish (US)
Title of host publicationMM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops
Pages103-112
Number of pages10
DOIs
StatePublished - 2011
Event19th ACM International Conference on Multimedia ACM Multimedia 2011, MM'11 - Scottsdale, AZ, United States
Duration: Nov 28 2011Dec 1 2011

Other

Other19th ACM International Conference on Multimedia ACM Multimedia 2011, MM'11
CountryUnited States
CityScottsdale, AZ
Period11/28/1112/1/11

Fingerprint

Image coding
Pixels
Cameras
Engines
Bandwidth
Experiments

Keywords

  • 3D image warping
  • Cloud gaming
  • Mobile devices
  • Real-time video coding

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction

Cite this

Shi, S., Hsu, C. H., Nahrstedt, K., & Campbell, R. H. (2011). Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming. In MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops (pp. 103-112) https://doi.org/10.1145/2072298.2072313

Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming. / Shi, Shu; Hsu, Cheng Hsin; Nahrstedt, Klara; Campbell, Roy H.

MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops. 2011. p. 103-112.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Shi, S, Hsu, CH, Nahrstedt, K & Campbell, RH 2011, Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming. in MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops. pp. 103-112, 19th ACM International Conference on Multimedia ACM Multimedia 2011, MM'11, Scottsdale, AZ, United States, 11/28/11. https://doi.org/10.1145/2072298.2072313
Shi S, Hsu CH, Nahrstedt K, Campbell RH. Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming. In MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops. 2011. p. 103-112 https://doi.org/10.1145/2072298.2072313
Shi, Shu ; Hsu, Cheng Hsin ; Nahrstedt, Klara ; Campbell, Roy H. / Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming. MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops. 2011. pp. 103-112
@inproceedings{1719e65d40a34bffb7d797f5e5c9b800,
title = "Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming",
abstract = "The emerging cloud gaming service has been growing rapidly, but not yet able to reach mobile customers due to many limitations, such as bandwidth and latency. We introduce a 3D image warping assisted real-time video coding method that can potentially meet all the requirements of mobile cloud gaming. The proposed video encoder selects a set of key frames in the video sequence, uses the 3D image warping algorithm to interpolate other non-key frames, and encodes the key frames and the residues frames with an H.264/AVC encoder. Our approach is novel in taking advantage of the run-time graphics rendering contexts (rendering viewpoint, pixel depth, camera motion, etc.) from the 3D game engine to enhance the performance of video encoding for the cloud gaming service. The experiments indicate that our proposed video encoder has the potential to beat the state-of-art x264 encoder in the scenario of real-time cloud gaming. For example, by implementing the proposed method in a 3D tank battle game, we experimentally show that more than 2 dB quality improvement is possible.",
keywords = "3D image warping, Cloud gaming, Mobile devices, Real-time video coding",
author = "Shu Shi and Hsu, {Cheng Hsin} and Klara Nahrstedt and Campbell, {Roy H.}",
year = "2011",
doi = "10.1145/2072298.2072313",
language = "English (US)",
isbn = "9781450306164",
pages = "103--112",
booktitle = "MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops",

}

TY - GEN

T1 - Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming

AU - Shi, Shu

AU - Hsu, Cheng Hsin

AU - Nahrstedt, Klara

AU - Campbell, Roy H.

PY - 2011

Y1 - 2011

N2 - The emerging cloud gaming service has been growing rapidly, but not yet able to reach mobile customers due to many limitations, such as bandwidth and latency. We introduce a 3D image warping assisted real-time video coding method that can potentially meet all the requirements of mobile cloud gaming. The proposed video encoder selects a set of key frames in the video sequence, uses the 3D image warping algorithm to interpolate other non-key frames, and encodes the key frames and the residues frames with an H.264/AVC encoder. Our approach is novel in taking advantage of the run-time graphics rendering contexts (rendering viewpoint, pixel depth, camera motion, etc.) from the 3D game engine to enhance the performance of video encoding for the cloud gaming service. The experiments indicate that our proposed video encoder has the potential to beat the state-of-art x264 encoder in the scenario of real-time cloud gaming. For example, by implementing the proposed method in a 3D tank battle game, we experimentally show that more than 2 dB quality improvement is possible.

AB - The emerging cloud gaming service has been growing rapidly, but not yet able to reach mobile customers due to many limitations, such as bandwidth and latency. We introduce a 3D image warping assisted real-time video coding method that can potentially meet all the requirements of mobile cloud gaming. The proposed video encoder selects a set of key frames in the video sequence, uses the 3D image warping algorithm to interpolate other non-key frames, and encodes the key frames and the residues frames with an H.264/AVC encoder. Our approach is novel in taking advantage of the run-time graphics rendering contexts (rendering viewpoint, pixel depth, camera motion, etc.) from the 3D game engine to enhance the performance of video encoding for the cloud gaming service. The experiments indicate that our proposed video encoder has the potential to beat the state-of-art x264 encoder in the scenario of real-time cloud gaming. For example, by implementing the proposed method in a 3D tank battle game, we experimentally show that more than 2 dB quality improvement is possible.

KW - 3D image warping

KW - Cloud gaming

KW - Mobile devices

KW - Real-time video coding

UR - http://www.scopus.com/inward/record.url?scp=84455173157&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84455173157&partnerID=8YFLogxK

U2 - 10.1145/2072298.2072313

DO - 10.1145/2072298.2072313

M3 - Conference contribution

AN - SCOPUS:84455173157

SN - 9781450306164

SP - 103

EP - 112

BT - MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops

ER -