Pruning via Iterative Ranking of Sensitivity Statistics

Abstract

With the introduction of SNIP [arXiv:1810.02340v2], it has been demonstrated that modern neural networks can effectively be pruned before training. Yet, its sensitivity criterion has since been criticized for not propagating training signal properly or even disconnecting layers. As a remedy, GraSP [arXiv:2002.07376v1] was introduced, compromising on simplicity. However, in this work we show that by applying the sensitivity criterion iteratively in smaller steps - still before training - we can improve its performance without difficult implementation. As such, we introduce 'SNIP-it'. We then demonstrate how it can be applied for both structured and unstructured pruning, before and/or during training, therewith achieving state-of-the-art sparsity-performance trade-offs. That is, while already providing the computational benefits of pruning in the training process from the start. Furthermore, we evaluate our methods on robustness to overfitting, disconnection and adversarial attacks as well.

25 pages, 21 figures, 62 pictures, typos corrected, reference added

References in corpus (31)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Do Deep Nets Really Need to be Deep?
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Pruning Filters for Efficient ConvNets
• Learning Structured Sparsity in Deep Neural Networks
• The State of Sparsity in Deep Neural Networks
• Channel Pruning for Accelerating Very Deep Neural Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Deep Rewiring: Training very sparse deep networks
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Stabilizing the Lottery Ticket Hypothesis
• Rigging the Lottery: Making All Tickets Winners
• Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• Stronger generalization bounds for deep nets via a compression approach
• Learning Sparse Networks Using Targeted Dropout
• Proving the Lottery Ticket Hypothesis: Pruning is All You Need
• Big Neural Networks Waste Capacity
• HYDRA: Pruning Adversarially Robust Neural Networks
• The Search for Sparse, Robust Neural Networks
• SS-Auto: A Single-Shot, Automatic Structured Weight Pruning Framework of DNNs with Ultra-High Efficiency
• Single-shot Channel Pruning Based on Alternating Direction Method of Multipliers
• Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training

Cited by in corpus (9)

• Pruning neural networks without any data by iteratively conserving synaptic flow
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• Progressive Skeletonization: Trimming more fat from a network at initialization
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• Sparse Linear Networks with a Fixed Butterfly Structure: Theory and Practice
• COPS: Controlled Pruning Before Training Starts
• ESPN: Extremely Sparse Pruned Networks
• Plant 'n' Seek: Can You Find the Winning Ticket?
• RGP: Neural Network Pruning through Its Regular Graph Structure