Compressing Recurrent Neural Network with Tensor Train

Abstract

Recurrent Neural Network (RNN) are a popular choice for modeling temporal and sequential tasks and achieve many state-of-the-art performance on various complex problems. However, most of the state-of-the-art RNNs have millions of parameters and require many computational resources for training and predicting new data. This paper proposes an alternative RNN model to reduce the number of parameters significantly by representing the weight parameters based on Tensor Train (TT) format. In this paper, we implement the TT-format representation for several RNN architectures such as simple RNN and Gated Recurrent Unit (GRU). We compare and evaluate our proposed RNN model with uncompressed RNN model on sequence classification and sequence prediction tasks. Our proposed RNNs with TT-format are able to preserve the performance while reducing the number of RNN parameters significantly up to 40 times smaller.

Accepted at IJCNN 2017

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Learning with Limited Numerical Precision
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

Cited by in corpus (12)

• Compression of Recurrent Neural Networks for Efficient Language Modeling
• Artificial neural networks condensation: A strategy to facilitate adaption of machine learning in medical settings by reducing computational burden
• How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection
• AdaDeep: A Usage-Driven, Automated Deep Model Compression Framework for Enabling Ubiquitous Intelligent Mobiles
• Convolutional Neural Networks with Transformed Input based on Robust Tensor Network Decomposition
• On the Memory Mechanism of Tensor-Power Recurrent Models
• Bayesian Sparsification of Gated Recurrent Neural Networks
• Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators
• Tensor train decompositions on recurrent networks
• Large-Scale Learning with Fourier Features and Tensor Decompositions