Training Quantized Neural Networks with a Full-precision Auxiliary Module
arXiv:1903.11236
Abstract
In this paper, we seek to tackle a challenge in training low-precision networks: the notorious difficulty in propagating gradient through a low-precision network due to the non-differentiable quantization function. We propose a solution by training the low-precision network with a fullprecision auxiliary module. Specifically, during training, we construct a mix-precision network by augmenting the original low-precision network with the full precision auxiliary module. Then the augmented mix-precision network and the low-precision network are jointly optimized. This strategy creates additional full-precision routes to update the parameters of the low-precision model, thus making the gradient back-propagates more easily. At the inference time, we discard the auxiliary module without introducing any computational complexity to the low-precision network. We evaluate the proposed method on image classification and object detection over various quantization approaches and show consistent performance increase. In particular, we achieve near lossless performance to the full-precision model by using a 4-bit detector, which is of great practical value.
arXiv commentAccepted to Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR 2020)
References in corpus (14)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Distilling the Knowledge in a Neural Network
- YOLOv3: An Incremental Improvement
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Binarized Neural Networks
- Trained Ternary Quantization
- Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
- Loss-aware Binarization of Deep Networks
- ProxQuant: Quantized Neural Networks via Proximal Operators
- Training Competitive Binary Neural Networks from Scratch
- Joint Neural Architecture Search and Quantization
- Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations