Compressing Recurrent Neural Network with Tensor Train
arXiv:1705.08052 · doi:10.1109/IJCNN.2017.7966420
Abstract
Recurrent Neural Network (RNN) are a popular choice for modeling temporal and sequential tasks and achieve many state-of-the-art performance on various complex problems. However, most of the state-of-the-art RNNs have millions of parameters and require many computational resources for training and predicting new data. This paper proposes an alternative RNN model to reduce the number of parameters significantly by representing the weight parameters based on Tensor Train (TT) format. In this paper, we implement the TT-format representation for several RNN architectures such as simple RNN and Gated Recurrent Unit (GRU). We compare and evaluate our proposed RNN model with uncompressed RNN model on sequence classification and sequence prediction tasks. Our proposed RNNs with TT-format are able to preserve the performance while reducing the number of RNN parameters significantly up to 40 times smaller.
Accepted at IJCNN 2017
References in corpus (7)
- Distilling the Knowledge in a Neural Network
- Sequence to Sequence Learning with Neural Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Deep Learning with Limited Numerical Precision
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
Cited by in corpus (5)
- Compression of Recurrent Neural Networks for Efficient Language Modeling
- Artificial neural networks condensation: A strategy to facilitate adaption of machine learning in medical settings by reducing computational burden
- How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection
- Convolutional Neural Networks with Transformed Input based on Robust Tensor Network Decomposition
- Bayesian Sparsification of Gated Recurrent Neural Networks