Pruning via Iterative Ranking of Sensitivity Statistics
arXiv:2006.00896
Abstract
With the introduction of SNIP [arXiv:1810.02340v2], it has been demonstrated that modern neural networks can effectively be pruned before training. Yet, its sensitivity criterion has since been criticized for not propagating training signal properly or even disconnecting layers. As a remedy, GraSP [arXiv:2002.07376v1] was introduced, compromising on simplicity. However, in this work we show that by applying the sensitivity criterion iteratively in smaller steps - still before training - we can improve its performance without difficult implementation. As such, we introduce 'SNIP-it'. We then demonstrate how it can be applied for both structured and unstructured pruning, before and/or during training, therewith achieving state-of-the-art sparsity-performance trade-offs. That is, while already providing the computational benefits of pruning in the training process from the start. Furthermore, we evaluate our methods on robustness to overfitting, disconnection and adversarial attacks as well.
25 pages, 21 figures, 62 pictures, typos corrected, reference added
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Pruning Filters for Efficient ConvNets
- Learning Structured Sparsity in Deep Neural Networks
- The State of Sparsity in Deep Neural Networks
- Channel Pruning for Accelerating Very Deep Neural Networks
- AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
- Proving the Lottery Ticket Hypothesis: Pruning is All You Need
- Big Neural Networks Waste Capacity
- The Search for Sparse, Robust Neural Networks
- SS-Auto: A Single-Shot, Automatic Structured Weight Pruning Framework of DNNs with Ultra-High Efficiency
- Single-shot Channel Pruning Based on Alternating Direction Method of Multipliers