AntMan: Sparse Low-Rank Compression to Accelerate RNN inference
arXiv:1910.01740
Abstract
Wide adoption of complex RNN based models is hindered by their inference performance, cost and memory requirements. To address this issue, we develop AntMan, combining structured sparsity with low-rank decomposition synergistically, to reduce model computation, size and execution time of RNNs while attaining desired accuracy. AntMan extends knowledge distillation based training to learn the compressed models efficiently. Our evaluation shows that AntMan offers up to 100x computation reduction with less than 1pt accuracy drop for language and machine reading comprehension models. Our evaluation also shows that for a given accuracy target, AntMan produces 5x smaller models than the state-of-art. Lastly, we show that AntMan offers super-linear speed gains compared to theoretical speedup, demonstrating its practical value on commodity hardware.
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- Neural Architecture Search with Reinforcement Learning
- Recurrent Neural Network Regularization
- Bidirectional Attention Flow for Machine Comprehension
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Regularizing and Optimizing LSTM Language Models
- Bayesian Dark Knowledge
- Exploring Sparsity in Recurrent Neural Networks
- Learning Intrinsic Sparse Structures within Long Short-Term Memory
- Block-Sparse Recurrent Neural Networks
- Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
- Retraining-Based Iterative Weight Quantization for Deep Neural Networks