GDRQ: Group-based Distribution Reshaping for Quantization
arXiv:1908.01477
Abstract
Low-bit quantization is challenging to maintain high performance with limited model capacity (e.g., 4-bit for both weights and activations). Naturally, the distribution of both weights and activations in deep neural network are Gaussian-like. Nevertheless, due to the limited bitwidth of low-bit model, uniform-like distributed weights and activations have been proved to be more friendly to quantization while preserving accuracy~\cite{Han2015Learning}. Motivated by this, we propose Scale-Clip, a Distribution Reshaping technique that can reshape weights or activations into a uniform-like distribution in a dynamic manner. Furthermore, to increase the model capability for a low-bit model, a novel Group-based Quantization algorithm is proposed to split the filters into several groups. Different groups can learn different quantization parameters, which can be elegantly merged in to batch normalization layer without extra computational cost in the inference stage. Finally, we integrate Scale-Clip technique with Group-based Quantization algorithm and propose the Group-based Distribution Reshaping Quantization (GDQR) framework to further improve the quantization performance. Experiments on various networks (e.g. VGGNet and ResNet) and vision tasks (e.g. classification, detection and segmentation) demonstrate that our framework achieves good performance.
References in corpus (14)
- Deep Learning with Differential Privacy
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Trained Ternary Quantization
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
- Convolutional Neural Networks using Logarithmic Data Representation
- Adding Gradient Noise Improves Learning for Very Deep Networks
- Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
- Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
- Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks
- Ternary Neural Networks with Fine-Grained Quantization
- Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets