Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition

Abstract

Recurrent Neural Networks (RNNs) are powerful sequence modeling tools. However, when dealing with high dimensional inputs, the training of RNNs becomes computational expensive due to the large number of model parameters. This hinders RNNs from solving many important computer vision tasks, such as Action Recognition in Videos and Image Captioning. To overcome this problem, we propose a compact and flexible structure, namely Block-Term tensor decomposition, which greatly reduces the parameters of RNNs and improves their training efficiency. Compared with alternative low-rank approximations, such as tensor-train RNN (TT-RNN), our method, Block-Term RNN (BT-RNN), is not only more concise (when using the same rank), but also able to attain a better approximation to the original RNNs with much fewer parameters. On three challenging tasks, including Action Recognition in Videos, Image Captioning and Image Generation, BT-RNN outperforms TT-RNN and the standard RNN in terms of both prediction accuracy and convergence rate. Specifically, BT-LSTM utilizes 17,388 times fewer parameters than the standard LSTM to achieve an accuracy improvement over 15.6\% in the Action Recognition task on the UCF11 dataset.

CVPR2018

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Densely Connected Convolutional Networks
• DRAW: A Recurrent Neural Network For Image Generation
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Compressing Recurrent Neural Network with Tensor Train
• Long-term Forecasting using Higher Order Tensor RNNs
• Tensor-Train Recurrent Neural Networks for Video Classification
• On the Compression of Recurrent Neural Networks with an Application to LVCSR acoustic modeling for Embedded Speech Recognition
• Sharing Residual Units Through Collective Tensor Factorization in Deep Neural Networks
• Learning Compact Recurrent Neural Networks
• Learning Efficient Tensor Representations with Ring Structure Networks
• BT-Nets: Simplifying Deep Neural Networks via Block Term Decomposition
• Tensorial Recurrent Neural Networks for Longitudinal Data Analysis
• Tensor Decomposition via Variational Auto-Encoder
• Simple and Efficient Parallelization for Probabilistic Temporal Tensor Factorization

Cited by in corpus (6)

• FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
• Hybrid Tensor Decomposition in Neural Network Compression
• Long short-term memory networks for proton dose calculation in highly heterogeneous tissues
• Tensor Decomposition for Compressing Recurrent Neural Network
• AntMan: Sparse Low-Rank Compression to Accelerate RNN inference
• SuperNeurons: FFT-based Gradient Sparsification in the Distributed Training of Deep Neural Networks