Bayesian Sparsification of Recurrent Neural Networks
arXiv:1708.00077
Abstract
Recurrent neural networks show state-of-the-art results in many text analysis tasks but often require a lot of memory to store their weights. Recently proposed Sparse Variational Dropout eliminates the majority of the weights in a feed-forward neural network without significant loss of quality. We apply this technique to sparsify recurrent neural networks. To account for recurrent specifics we also rely on Binary Variational Dropout for RNN. We report 99.5% sparsity level on sentiment analysis task without a quality drop and up to 87% sparsity level on language modeling task with slight loss of accuracy.
arXiv commentPublished in Workshop on Learning to Generate Natural Language, ICML, 2017
References in corpus (13)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
- Recurrent Neural Network Regularization
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
- Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
- Bayesian Compression for Deep Learning
- Deep Networks with Stochastic Depth
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
- Exploring Sparsity in Recurrent Neural Networks
- Structured Bayesian Pruning via Log-Normal Multiplicative Noise
- Recurrent Dropout without Memory Loss
- On Fast Dropout and its Applicability to Recurrent Networks
Cited by in corpus (6)
- Tensorized Embedding Layers for Efficient Model Compression
- Compression of Recurrent Neural Networks for Efficient Language Modeling
- One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation
- Intrinsically Sparse Long Short-Term Memory Networks
- GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking
- Spectral Pruning for Recurrent Neural Networks