Toward Extremely Low Bit and Lossless Accuracy in DNNs with Progressive ADMM
arXiv:1905.00789
Abstract
Weight quantization is one of the most important techniques of Deep Neural Networks (DNNs) model compression method. A recent work using systematic framework of DNN weight quantization with the advanced optimization algorithm ADMM (Alternating Direction Methods of Multipliers) achieves one of state-of-art results in weight quantization. In this work, we first extend such ADMM-based framework to guarantee solution feasibility and we have further developed a multi-step, progressive DNN weight quantization framework, with dual benefits of (i) achieving further weight quantization thanks to the special property of ADMM regularization, and (ii) reducing the search space within each step. Extensive experimental results demonstrate the superior performance compared with prior work. Some highlights: we derive the first lossless and fully binarized (for all layers) LeNet-5 for MNIST; And we derive the first fully binarized (for all layers) VGG-16 for CIFAR-10 and ResNet for ImageNet with reasonable accuracy loss.
arXiv commentAccepted by ICML workshop (ODML-CDNNR2019). arXiv admin note: substantial text overlap with arXiv:1903.09769
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Binarized Neural Networks
- Fixed Point Quantization of Deep Convolutional Networks
- A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
- Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
- Differentiable Fine-grained Quantization for Deep Neural Network Compression
Cited by in corpus (5)
- A SOT-MRAM-based Processing-In-Memory Engine for Highly Compressed DNN Implementation
- Tiny but Accurate: A Pruned, Quantized and Optimized Memristor Crossbar Framework for Ultra Efficient DNN Implementation
- Teachers Do More Than Teach: Compressing Image-to-Image Models
- An Ultra-Efficient Memristor-Based DNN Framework with Structured Weight Pruning and Quantization Using ADMM
- Alternating Direction Method of Multipliers for Quantization