SS-Auto: A Single-Shot, Automatic Structured Weight Pruning Framework of DNNs with Ultra-High Efficiency
arXiv:2001.08839
Abstract
Structured weight pruning is a representative model compression technique of DNNs for hardware efficiency and inference accelerations. Previous works in this area leave great space for improvement since sparse structures with combinations of different structured pruning schemes are not exploited fully and efficiently. To mitigate the limitations, we propose SS-Auto, a single-shot, automatic structured pruning framework that can achieve row pruning and column pruning simultaneously. We adopt soft constraint-based formulation to alleviate the strong non-convexity of l0-norm constraints used in state-of-the-art ADMM-based methods for faster convergence and fewer hyperparameters. Instead of solving the problem directly, a Primal-Proximal solution is proposed to avoid the pitfall of penalizing all weights equally, thereby enhancing the accuracy. Extensive experiments on CIFAR-10 and CIFAR-100 datasets demonstrate that the proposed framework can achieve ultra-high pruning rates while maintaining accuracy. Furthermore, significant inference speedup has been observed from the proposed framework through actual measurements on the smartphone.
References in corpus (9)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- Pruning Filters for Efficient ConvNets
- Large-Scale Evolution of Image Classifiers
- Discrimination-aware Channel Pruning for Deep Neural Networks
- N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
- 2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy
- Systematic Weight Pruning of DNNs using Alternating Direction Method of Multipliers