Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMM

Abstract

Weight pruning and weight quantization are two important categories of DNN model compression. Prior work on these techniques are mainly based on heuristics. A recent work developed a systematic frame-work of DNN weight pruning using the advanced optimization technique ADMM (Alternating Direction Methods of Multipliers), achieving one of state-of-art in weight pruning results. In this work, we first extend such one-shot ADMM-based framework to guarantee solution feasibility and provide fast convergence rate, and generalize to weight quantization as well. We have further developed a multi-step, progressive DNN weight pruning and quantization framework, with dual benefits of (i) achieving further weight pruning/quantization thanks to the special property of ADMM regularization, and (ii) reducing the search space within each step. Extensive experimental results demonstrate the superior performance compared with prior work. Some highlights: (i) we achieve 246x,36x, and 8x weight pruning on LeNet-5, AlexNet, and ResNet-50 models, respectively, with (almost) zero accuracy loss; (ii) even a significant 61x weight pruning in AlexNet (ImageNet) results in only minor degradation in actual accuracy compared with prior work; (iii) we are among the first to derive notable weight pruning results for ResNet and MobileNet models; (iv) we derive the first lossless, fully binarized (for all layers) LeNet-5 for MNIST and VGG-16 for CIFAR-10; and (v) we derive the first fully binarized (for all layers) ResNet for ImageNet with reasonable accuracy loss.

References in corpus (8)

• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Binarized Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
• Differentiable Fine-grained Quantization for Deep Neural Network Compression
• Demystifying Neural Network Filter Pruning

Cited by in corpus (14)

• Learned Threshold Pruning
• 26ms Inference Time for ResNet-50: Towards Real-Time Execution of all DNNs on Smartphone
• Mirror Descent View for Neural Network Quantization
• AlphaGAN: Fully Differentiable Architecture Search for Generative Adversarial Networks
• Tiny but Accurate: A Pruned, Quantized and Optimized Memristor Crossbar Framework for Ultra Efficient DNN Implementation
• An Ultra-Efficient Memristor-Based DNN Framework with Structured Weight Pruning and Quantization Using ADMM
• ResNet Can Be Pruned 60x: Introducing Network Purification and Unused Path Removal (P-RM) after Weight Pruning
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• AUSN: Approximately Uniform Quantization by Adaptively Superimposing Non-uniform Distribution for Deep Neural Networks
• A Unified DNN Weight Compression Framework Using Reweighted Optimization Methods
• Alternating Direction Method of Multipliers for Quantization
• ADMP: An Adversarial Double Masks Based Pruning Framework For Unsupervised Cross-Domain Compression
• Active Subspace of Neural Networks: Structural Analysis and Universal Attacks
• Cascade Weight Shedding in Deep Neural Networks: Benefits and Pitfalls for Network Pruning