Self-Reproducing Video Frame Interpolation

Jiajun Deng, Haichao Yu, Zhangyang Wang, Xinchao Wang, Thomas S Huang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Frame interpolation has recently witnessed success by convolutional neural networks, that are learned from end to end to minimizing the reconstruction loss of dropped frames. This paper introduces a novel self-reproducing mechanism, that the real (given) frames could in turn be interpolated from the interpolated ones, to further substantially improve the consistency and performance of video frame interpolation. Such a consistency constraint accounts for the inherent symmetry between existing and interpolated frames in a video sequence, providing a strong form of self-supervision. We then build a pyramid-like architecture, under which existing interpolation models can plug-and-play as building blocks. Extensive experiments validate its state-of-the-art performance, on both high resolution videos in the wild and public benchmarks.

Original languageEnglish (US)
Title of host publicationProceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages193-198
Number of pages6
ISBN (Electronic)9781728111988
DOIs
StatePublished - Apr 22 2019
Event2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019 - San Jose, United States
Duration: Mar 28 2019Mar 30 2019

Publication series

NameProceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019

Conference

Conference2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019
CountryUnited States
CitySan Jose
Period3/28/193/30/19

Fingerprint

Interpolation
Neural networks
Experiments

Keywords

  • Deep Learning
  • Frame interpolation
  • Self supervision

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Signal Processing
  • Media Technology

Cite this

Deng, J., Yu, H., Wang, Z., Wang, X., & Huang, T. S. (2019). Self-Reproducing Video Frame Interpolation. In Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019 (pp. 193-198). [8695418] (Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/MIPR.2019.00042

Self-Reproducing Video Frame Interpolation. / Deng, Jiajun; Yu, Haichao; Wang, Zhangyang; Wang, Xinchao; Huang, Thomas S.

Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 193-198 8695418 (Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Deng, J, Yu, H, Wang, Z, Wang, X & Huang, TS 2019, Self-Reproducing Video Frame Interpolation. in Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019., 8695418, Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019, Institute of Electrical and Electronics Engineers Inc., pp. 193-198, 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019, San Jose, United States, 3/28/19. https://doi.org/10.1109/MIPR.2019.00042
Deng J, Yu H, Wang Z, Wang X, Huang TS. Self-Reproducing Video Frame Interpolation. In Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 193-198. 8695418. (Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019). https://doi.org/10.1109/MIPR.2019.00042
Deng, Jiajun ; Yu, Haichao ; Wang, Zhangyang ; Wang, Xinchao ; Huang, Thomas S. / Self-Reproducing Video Frame Interpolation. Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 193-198 (Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019).
@inproceedings{da6e6c118cf04351b187d5ae97ae3c76,
title = "Self-Reproducing Video Frame Interpolation",
abstract = "Frame interpolation has recently witnessed success by convolutional neural networks, that are learned from end to end to minimizing the reconstruction loss of dropped frames. This paper introduces a novel self-reproducing mechanism, that the real (given) frames could in turn be interpolated from the interpolated ones, to further substantially improve the consistency and performance of video frame interpolation. Such a consistency constraint accounts for the inherent symmetry between existing and interpolated frames in a video sequence, providing a strong form of self-supervision. We then build a pyramid-like architecture, under which existing interpolation models can plug-and-play as building blocks. Extensive experiments validate its state-of-the-art performance, on both high resolution videos in the wild and public benchmarks.",
keywords = "Deep Learning, Frame interpolation, Self supervision",
author = "Jiajun Deng and Haichao Yu and Zhangyang Wang and Xinchao Wang and Huang, {Thomas S}",
year = "2019",
month = "4",
day = "22",
doi = "10.1109/MIPR.2019.00042",
language = "English (US)",
series = "Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "193--198",
booktitle = "Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019",
address = "United States",

}

TY - GEN

T1 - Self-Reproducing Video Frame Interpolation

AU - Deng, Jiajun

AU - Yu, Haichao

AU - Wang, Zhangyang

AU - Wang, Xinchao

AU - Huang, Thomas S

PY - 2019/4/22

Y1 - 2019/4/22

N2 - Frame interpolation has recently witnessed success by convolutional neural networks, that are learned from end to end to minimizing the reconstruction loss of dropped frames. This paper introduces a novel self-reproducing mechanism, that the real (given) frames could in turn be interpolated from the interpolated ones, to further substantially improve the consistency and performance of video frame interpolation. Such a consistency constraint accounts for the inherent symmetry between existing and interpolated frames in a video sequence, providing a strong form of self-supervision. We then build a pyramid-like architecture, under which existing interpolation models can plug-and-play as building blocks. Extensive experiments validate its state-of-the-art performance, on both high resolution videos in the wild and public benchmarks.

AB - Frame interpolation has recently witnessed success by convolutional neural networks, that are learned from end to end to minimizing the reconstruction loss of dropped frames. This paper introduces a novel self-reproducing mechanism, that the real (given) frames could in turn be interpolated from the interpolated ones, to further substantially improve the consistency and performance of video frame interpolation. Such a consistency constraint accounts for the inherent symmetry between existing and interpolated frames in a video sequence, providing a strong form of self-supervision. We then build a pyramid-like architecture, under which existing interpolation models can plug-and-play as building blocks. Extensive experiments validate its state-of-the-art performance, on both high resolution videos in the wild and public benchmarks.

KW - Deep Learning

KW - Frame interpolation

KW - Self supervision

UR - http://www.scopus.com/inward/record.url?scp=85065612405&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065612405&partnerID=8YFLogxK

U2 - 10.1109/MIPR.2019.00042

DO - 10.1109/MIPR.2019.00042

M3 - Conference contribution

AN - SCOPUS:85065612405

T3 - Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019

SP - 193

EP - 198

BT - Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019

PB - Institute of Electrical and Electronics Engineers Inc.

ER -