Convolutional Tensor-Train LSTM for Spatio-temporal Learning

Abstract

Learning from spatio-temporal data has numerous applications such as human-behavior analysis, object tracking, video compression, and physics simulation.However, existing methods still perform poorly on challenging video tasks such as long-term forecasting. This is because these kinds of challenging tasks require learning long-term spatio-temporal correlations in the video sequence. In this paper, we propose a higher-order convolutional LSTM model that can efficiently learn these correlations, along with a succinct representations of the history. This is accomplished through a novel tensor train module that performs prediction by combining convolutional features across time. To make this feasible in terms of computation and memory requirements, we propose a novel convolutional tensor-train decomposition of the higher-order model. This decomposition reduces the model complexity by jointly approximating a sequence of convolutional kernels asa low-rank tensor-train factorization. As a result, our model outperforms existing approaches, but uses only a fraction of parameters, including the baseline models.Our results achieve state-of-the-art performance in a wide range of applications and datasets, including the multi-steps video prediction on the Moving-MNIST-2and KTH action datasets as well as early activity recognition on the Something-Something V2 dataset.

Jiahao Su and Wonmin Byeon contributed equally to this work. 22 pages, 14 figures, NeurIPS 2020

References in corpus (25)

• Adam: A Method for Stochastic Optimization
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Unsupervised Learning of Video Representations using LSTMs
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Tensorizing Neural Networks
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Stochastic Video Generation with a Learned Prior
• Image Transformer
• Stochastic Adversarial Video Prediction
• Compressing Recurrent Neural Network with Tensor Train
• Architectural Complexity Measures of Recurrent Neural Networks
• Deep Multi-task Representation Learning: A Tensor Factorisation Approach
• A Tensorized Transformer for Language Modeling
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Tensor Regression Networks
• Long-term Forecasting using Higher Order Tensor RNNs
• Stochastic Variational Video Prediction
• Tensor-Train Recurrent Neural Networks for Video Classification
• Higher Order Recurrent Neural Networks
• Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
• Scaling Autoregressive Video Models

Cited by in corpus (8)

• Tensor Methods in Computer Vision and Deep Learning
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Advancing Learned Video Compression with In-loop Frame Prediction
• Tensor networks for interpretable and efficient quantum-inspired machine learning
• MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions
• Performing Video Frame Prediction of Microbial Growth with a Recurrent Neural Network
• FDNet: A Deep Learning Approach with Two Parallel Cross Encoding Pathways for Precipitation Nowcasting
• On the Memory Mechanism of Tensor-Power Recurrent Models