Stochastic Model Pruning via Weight Dropping Away and Back
arXiv:1812.02035
Abstract
Deep neural networks have dramatically achieved great success on a variety of challenging tasks. However, most successful DNNs have an extremely complex structure, leading to extensive research on model compression.As a significant area of progress in model compression, traditional gradual pruning approaches involve an iterative prune-retrain procedure and may suffer from two critical issues: local importance judgment, where the pruned weights are merely unimportant in the current model; and an irretrievable pruning process, where the pruned weights have no chance to come back. Addressing these two issues, this paper proposes the Drop Pruning approach, which leverages stochastic optimization in the pruning process by introducing a drop strategy at each pruning step, namely, drop away, which stochastically deletes some unimportant weights, and drop back, which stochastically recovers some pruned weights. The suitable choice of drop probabilities decreases the model size during the pruning process and helps it flow to the target sparsity. Compared to the Bayesian approaches that stochastically train a compact model for pruning, we directly aim at stochastic gradual pruning. We provide a detailed analysis showing that the drop away and drop back approaches have individual contributions. Moreover, Drop Pruning can achieve competitive compression performance and accuracy on many benchmark tasks compared with state-of-the-art weights pruning and Bayesian training approaches.
References in corpus (15)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Improving neural networks by preventing co-adaptation of feature detectors
- Do Deep Nets Really Need to be Deep?
- An Analysis of Deep Neural Network Models for Practical Applications
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- To prune, or not to prune: exploring the efficacy of pruning for model compression
- Compressing Neural Networks with the Hashing Trick
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- Variational Dropout and the Local Reparameterization Trick
- DSD: Dense-Sparse-Dense Training for Deep Neural Networks
- Structured Bayesian Pruning via Log-Normal Multiplicative Noise
- Compressing Neural Networks using the Variational Information Bottleneck
- Dropout as data augmentation
- DeepTwist: Learning Model Compression via Occasional Weight Distortion
- Full deep neural network training on a pruned weight budget