Tensor-Train Recurrent Neural Networks for Video Classification

Abstract

The Recurrent Neural Networks and their variants have shown promising performances in sequence modeling tasks such as Natural Language Processing. These models, however, turn out to be impractical and difficult to train when exposed to very high-dimensional inputs due to the large input-to-hidden weight matrix. This may have prevented RNNs' large-scale application in tasks that involve very high input dimensions such as video modeling; current approaches reduce the input dimensions using various feature extractors. To address this challenge, we propose a new, more general and efficient approach by factorizing the input-to-hidden weight matrix using Tensor-Train decomposition which is trained simultaneously with the weights themselves. We test our model on classification tasks using multiple real-world video datasets and achieve competitive performances with state-of-the-art models, even though our model architecture is orders of magnitude less complex. We believe that the proposed approach provides a novel and fundamental building block for modeling high-dimensional sequential data with RNN architectures and opens up many possibilities to transfer the expressive and advanced architectures from other domains such as NLP to modeling high-dimensional sequential data.

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Theano: new features and speed improvements
• Tensorizing Neural Networks
• Action Recognition using Visual Attention
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Ultimate tensorization: compressing convolutional and FC layers alike

Cited by in corpus (32)

• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Long-term Forecasting using Higher Order Tensor RNNs
• Tensorized Embedding Layers for Efficient Model Compression
• Hybrid Tensor Decomposition in Neural Network Compression
• Tensor Train decomposition on TensorFlow (T3F)
• Multilinear Compressive Learning
• Kronecker CP Decomposition with Fast Multiplication for Compressing RNNs
• Wide Compression: Tensor Ring Nets
• On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression
• Compressing 3DCNNs Based on Tensor Train Decomposition
• Connecting Weighted Automata and Recurrent Neural Networks through Spectral Learning
• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
• A Model Compression Method with Matrix Product Operators for Speech Enhancement
• TedNet: A Pytorch Toolkit for Tensor Decomposition Networks
• QTN-VQC: An End-to-End Learning framework for Quantum Neural Networks
• Sparse Tucker Tensor Decomposition on a Hybrid FPGA-CPU Platform
• Structured Convolutions for Efficient Neural Network Design
• Adaptive Learning of Tensor Network Structures
• DEEPEYE: A Compact and Accurate Video Comprehension at Terminal Devices Compressed with Quantization and Tensorization
• Dynamic Sparse Graph for Efficient Deep Learning
• Dilated Convolutional Neural Networks for Sequential Manifold-valued Data
• Compressing LSTM Networks by Matrix Product Operators
• On the Memory Mechanism of Tensor-Power Recurrent Models
• A Fully Tensorized Recurrent Neural Network
• S3-Net: A Fast and Lightweight Video Scene Understanding Network by Single-shot Segmentation
• Block-term Tensor Neural Networks
• Neural network compression via learnable wavelet transforms
• Spectral Tensor Train Parameterization of Deep Learning Layers
• A Sampling-Based Method for Tensor Ring Decomposition
• Recurrent Graph Tensor Networks: A Low-Complexity Framework for Modelling High-Dimensional Multi-Way Sequence