Using Deep Convolutional LSTM Networks for Learning Spatiotemporal Features

Logan Courtney, Ramavarapu Sreenivas

Research output: Chapter in Book/Report/Conference proceedingConference contribution


This paper explores the use of convolutional LSTMs to simultaneously learn spatial- and temporal-information in videos. A deep network of convolutional LSTMs allows the model to access the entire range of temporal information at all spatial scales. We describe our experiments involving convolutional LSTMs for lipreading that demonstrate the model is capable of selectively choosing which spatiotemporal scales are most relevant for a particular dataset. The proposed deep architecture holds promise in other applications where spatiotemporal features play a vital role without having to specifically cater the design of the network for the particular spatiotemporal features existent within the problem. Our model has comparable performance with the current state of the art achieving 83.4% on the Lip Reading in the Wild (LRW) dataset. Additional experiments indicate convolutional LSTMs may be particularly data hungry considering the large performance increases when fine-tuning on LRW after pretraining on larger datasets like LRS2 (85.2%) and LRS3-TED (87.1%). However, a sensitivity analysis providing insight on the relevant spatiotemporal temporal features allows certain convolutional LSTM layers to be replaced with 2D convolutions decreasing computational cost without performance degradation indicating their usefulness in accelerating the architecture design process when approaching new problems.

Original languageEnglish (US)
Title of host publicationPattern Recognition - 5th Asian Conference, ACPR 2019, Revised Selected Papers
EditorsShivakumara Palaiahnakote, Gabriella Sanniti di Baja, Liang Wang, Wei Qi Yan
Number of pages14
ISBN (Print)9783030412982
StatePublished - 2020
Event5th Asian Conference on Pattern Recognition, ACPR 2019 - Auckland, New Zealand
Duration: Nov 26 2019Nov 29 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12047 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference5th Asian Conference on Pattern Recognition, ACPR 2019
Country/TerritoryNew Zealand


  • Action recognition
  • Convolutional LSTMs
  • Deep learning
  • Lip Reading
  • Video analytics

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)


Dive into the research topics of 'Using Deep Convolutional LSTM Networks for Learning Spatiotemporal Features'. Together they form a unique fingerprint.

Cite this