Neural Pruning via Growing Regularization
arXiv:2012.09243
Abstract
Regularization has long been utilized to learn sparsity in deep neural network pruning. However, its role is mainly explored in the small penalty strength regime. In this work, we extend its application to a new scenario where the regularization grows large gradually to tackle two central problems of pruning: pruning schedule and weight importance scoring. (1) The former topic is newly brought up in this work, which we find critical to the pruning performance while receives little research attention. Specifically, we propose an L2 regularization variant with rising penalty factors and show it can bring significant accuracy gains compared with its one-shot counterpart, even when the same weights are removed. (2) The growing penalty scheme also brings us an approach to exploit the Hessian information for more accurate pruning without knowing their specific values, thus not bothered by the common Hessian approximation problems. Empirically, the proposed algorithms are easy to implement and scalable to large datasets and networks in both structured and unstructured pruning. Their effectiveness is demonstrated with modern deep neural networks on the CIFAR and ImageNet datasets, achieving competitive results compared to many state-of-the-art algorithms. Our code and trained models are publicly available at https://github.com/mingsuntse/regularization-pruning.
arXiv commentAccepted by ICLR 2021
References in corpus (17)
- Deep Learning in Neural Networks: An Overview
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- Rethinking the Value of Network Pruning
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- The State of Sparsity in Deep Neural Networks
- Variational Dropout Sparsifies Deep Neural Networks
- Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
- Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
- Dynamic Model Pruning with Feedback
- Approximated Oracle Filter Pruning for Destructive CNN Width Optimization
- WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
- EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis