Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning
arXiv:2002.04809
Abstract
Magnitude-based pruning is one of the simplest methods for pruning neural networks. Despite its simplicity, magnitude-based pruning and its variants demonstrated remarkable performances for pruning modern architectures. Based on the observation that magnitude-based pruning indeed minimizes the Frobenius distortion of a linear operator corresponding to a single layer, we develop a simple pruning method, coined lookahead pruning, by extending the single layer optimization to a multi-layer optimization. Our experimental results demonstrate that the proposed method consistently outperforms magnitude-based pruning on various networks, including VGG and ResNet, particularly in the high-sparsity regime. See https://github.com/alinlab/lookahead_pruning for codes.
arXiv commentICLR 2020, camera ready
References in corpus (7)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- Rethinking the Value of Network Pruning
- To prune, or not to prune: exploring the efficacy of pruning for model compression
- The State of Sparsity in Deep Neural Networks
- Bayesian Compression for Deep Learning
- On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
Cited by in corpus (9)
- Pruning neural networks without any data by iteratively conserving synaptic flow
- Layer-adaptive sparsity for the Magnitude-based Pruning
- Task dependent Deep LDA pruning of neural networks
- F3-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis
- DKM: Differentiable K-Means Clustering Layer for Neural Network Compression
- ACP: Automatic Channel Pruning via Clustering and Swarm Intelligence Optimization for CNN
- AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning
- Pruning with Compensation: Efficient Channel Pruning for Deep Convolutional Neural Networks
- Coarse and fine-grained automatic cropping deep convolutional neural network