High-performance video content recognition with long-term recurrent convolutional network for FPGA

Xiaofan Zhang, Xinheng Liu, Anand Ramachandran, Chuanhao Zhuge, Shibin Tang, Peng Ouyang, Zuofu Cheng, Kyle Rupnow, Deming Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

FPGA is a promising candidate for the acceleration of Deep Neural Networks (DNN) with improved latency and energy consumption compared to CPU and GPU-based implementations. DNNs use sequences of layers of regular computation that are well suited for HLS-based design for FPGA. However, optimizing large neural networks under resource constraints is still a key challenge. HLS must manage on-chip computation, buffering resources, and off-chip memory accesses to minimize the total latency. In this paper, we present a design framework for DNNs that uses highly configurable IPs for neural network layers together with a new design space exploration engine for Resource Allocation Management (REALM). We also carry out efficient memory subsystem design and fixed-point weight re-training to further improve our FPGA solution. We demonstrate our design framework on the Long-term Recurrent Convolution Network for video inputs. Our implementation on a Xilinx VC709 board achieves 3.1X speedup compared to an NVIDIA K80 and 4.75X speedup compared to an Intel Xeon with 17.5X lower energy per image.

Original languageEnglish (US)
Title of host publication2017 27th International Conference on Field Programmable Logic and Applications, FPL 2017
EditorsDiana Gohringer, Dirk Stroobandt, Nele Mentens, Marco Santambrogio, Jari Nurmi
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9789090304281
DOIs
StatePublished - Oct 2 2017
Event27th International Conference on Field Programmable Logic and Applications, FPL 2017 - Gent, Belgium
Duration: Sep 4 2017Sep 6 2017

Publication series

Name2017 27th International Conference on Field Programmable Logic and Applications, FPL 2017

Other

Other27th International Conference on Field Programmable Logic and Applications, FPL 2017
CountryBelgium
CityGent
Period9/4/179/6/17

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Software

Fingerprint Dive into the research topics of 'High-performance video content recognition with long-term recurrent convolutional network for FPGA'. Together they form a unique fingerprint.

  • Cite this

    Zhang, X., Liu, X., Ramachandran, A., Zhuge, C., Tang, S., Ouyang, P., Cheng, Z., Rupnow, K., & Chen, D. (2017). High-performance video content recognition with long-term recurrent convolutional network for FPGA. In D. Gohringer, D. Stroobandt, N. Mentens, M. Santambrogio, & J. Nurmi (Eds.), 2017 27th International Conference on Field Programmable Logic and Applications, FPL 2017 [8056833] (2017 27th International Conference on Field Programmable Logic and Applications, FPL 2017). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.23919/FPL.2017.8056833