Self-Reproducing Video Frame Interpolation

Jiajun Deng, Haichao Yu, Zhangyang Wang, Xinchao Wang, Thomas Huang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Frame interpolation has recently witnessed success by convolutional neural networks, that are learned from end to end to minimizing the reconstruction loss of dropped frames. This paper introduces a novel self-reproducing mechanism, that the real (given) frames could in turn be interpolated from the interpolated ones, to further substantially improve the consistency and performance of video frame interpolation. Such a consistency constraint accounts for the inherent symmetry between existing and interpolated frames in a video sequence, providing a strong form of self-supervision. We then build a pyramid-like architecture, under which existing interpolation models can plug-and-play as building blocks. Extensive experiments validate its state-of-the-art performance, on both high resolution videos in the wild and public benchmarks.

Original languageEnglish (US)
Title of host publicationProceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages193-198
Number of pages6
ISBN (Electronic)9781728111988
DOIs
StatePublished - Apr 22 2019
Event2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019 - San Jose, United States
Duration: Mar 28 2019Mar 30 2019

Publication series

NameProceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019

Conference

Conference2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019
CountryUnited States
CitySan Jose
Period3/28/193/30/19

    Fingerprint

Keywords

  • Deep Learning
  • Frame interpolation
  • Self supervision

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Signal Processing
  • Media Technology

Cite this

Deng, J., Yu, H., Wang, Z., Wang, X., & Huang, T. (2019). Self-Reproducing Video Frame Interpolation. In Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019 (pp. 193-198). [8695418] (Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/MIPR.2019.00042