Layer-adaptive sparsity for the Magnitude-based Pruning

Abstract

Recent discoveries on neural network pruning reveal that, with a carefully chosen layerwise sparsity, a simple magnitude-based pruning achieves state-of-the-art tradeoff between sparsity and performance. However, without a clear consensus on "how to choose," the layerwise sparsities are mostly selected algorithm-by-algorithm, often resorting to handcrafted heuristics or an extensive hyperparameter search. To fill this gap, we propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score; the score is a rescaled version of weight magnitude that incorporates the model-level ${\ell }_2$ distortion incurred by pruning, and does not require any hyperparameter tuning or heavy computation. Under various image classification setups, LAMP consistently outperforms popular existing schemes for layerwise sparsity selection. Furthermore, we observe that LAMP continues to outperform baselines even in weight-rewinding setups, while the connectivity-oriented layerwise sparsity (the strongest baseline overall) performs worse than a simple global magnitude-based pruning in this case. Code: https://github.com/jaeho-lee/layer-adaptive-sparsity

ICLR 2021. Changed title (previous ver: A deeper look at the layerwise sparsity of magnitude-based pruning)

References in corpus (17)

• Decoupled Weight Decay Regularization
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Rethinking the Value of Network Pruning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• The State of Sparsity in Deep Neural Networks
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• The Singular Values of Convolutional Layers
• Compressing Neural Networks using the Variational Information Bottleneck
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Dynamic Model Pruning with Feedback
• Norm-Based Capacity Control in Neural Networks
• Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning

Cited by in corpus (2)

• Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation