Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression
arXiv:2003.08935
Abstract
In this paper, we analyze two popular network compression techniques, i.e. filter pruning and low-rank decomposition, in a unified sense. By simply changing the way the sparsity regularization is enforced, filter pruning and low-rank decomposition can be derived accordingly. This provides another flexible choice for network compression because the techniques complement each other. For example, in popular network architectures with shortcut connections (e.g. ResNet), filter pruning cannot deal with the last convolutional layer in a ResBlock while the low-rank decomposition methods can. In addition, we propose to compress the whole network jointly instead of in a layer-wise manner. Our approach proves its potential as it compares favorably to the state-of-the-art on several benchmarks.
arXiv commentAccepted by CVPR2020. Code is available at https://github.com/ofsoundof/group_sparsity
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- Learning the Number of Neurons in Deep Networks
- Compression-aware Training of Deep Networks
- N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
- Moonshine: Distilling with Cheap Convolutions
- GASL: Guided Attention for Sparsity Learning in Deep Neural Networks
- Accelerate CNN via Recursive Bayesian Pruning
- Fast Learning with Nonconvex L1-2 Regularization