Image Captioning with Sparse Recurrent Neural Network
arXiv:1908.10797
Abstract
Recurrent Neural Network (RNN) has been widely used to tackle a wide variety of language generation problems and are capable of attaining state-of-the-art (SOTA) performance. However despite its impressive results, the large number of parameters in the RNN model makes deployment to mobile and embedded devices infeasible. Driven by this problem, many works have proposed a number of pruning methods to reduce the sizes of the RNN model. In this work, we propose an end-to-end pruning method for image captioning models equipped with visual attention. Our proposed method is able to achieve sparsity levels up to 97.5% without significant performance loss relative to the baseline (~ 2% loss at 40x compression after fine-tuning). Our method is also simple to use and tune, facilitating faster development times for neural network practitioners. We perform extensive experiments on the popular MS-COCO dataset in order to empirically validate the efficacy of our proposed method.
arXiv commentCorrected Eq 11, updated Table 5
References in corpus (17)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Distilling the Knowledge in a Neural Network
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- Understanding deep learning requires rethinking generalization
- To prune, or not to prune: exploring the efficacy of pruning for model compression
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- Variational Dropout and the Local Reparameterization Trick
- Variational Dropout Sparsifies Deep Neural Networks
- Exploring Sparsity in Recurrent Neural Networks
- Compressing Neural Networks using the Variational Information Bottleneck
- Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
- Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
- LightRNN: Memory and Computation-Efficient Recurrent Neural Networks
- Grow and Prune Compact, Fast, and Accurate LSTMs