Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks

Abstract

The quantized neural networks (QNNs) can be useful for neural network acceleration and compression, but during the training process they pose a challenge: how to propagate the gradient of loss function through the graph flow with a derivative of 0 almost everywhere. In response to this non-differentiable situation, we propose a novel Asymptotic-Quantized Estimator (AQE) to estimate the gradient. In particular, during back-propagation, the graph that relates inputs to output remains smoothness and differentiability. At the end of training, the weights and activations have been quantized to low-precision because of the asymptotic behaviour of AQE. Meanwhile, we propose a M-bit Inputs and N-bit Weights Network (MINW-Net) trained by AQE, a quantized neural network with 1-3 bits weights and activations. In the inference phase, we can use XNOR or SHIFT operations instead of convolution operations to accelerate the MINW-Net. Our experiments on CIFAR datasets demonstrate that our AQE is well defined, and the QNNs with AQE perform better than that with Straight-Through Estimator (STE). For example, in the case of the same ConvNet that has 1-bit weights and activations, our MINW-Net with AQE can achieve a prediction accuracy 1.5\% higher than the Binarized Neural Network (BNN) with STE. The MINW-Net, which is trained from scratch by AQE, can achieve comparable classification accuracy as 32-bit counterparts on CIFAR test sets. Extensive experimental results on ImageNet dataset show great superiority of the proposed AQE and our MINW-Net achieves comparable results with other state-of-the-art QNNs.

This paper has been accepted for publication in the IEEE Journal of Selected Topics in Signal Processing

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Fully Convolutional Networks for Semantic Segmentation
• Striving for Simplicity: The All Convolutional Net
• Going Deeper with Convolutions
• Deep Learning with Limited Numerical Precision
• Trained Ternary Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Towards Accurate Binary Convolutional Neural Network
• Gradient learning in spiking neural networks by dynamic perturbation of conductances
• Ternary Neural Networks with Fine-Grained Quantization
• Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)
• High performance ultra-low-precision convolutions on mobile devices
• Relaxed Quantization for Discretized Neural Networks
• Precision Highway for Ultra Low-Precision Quantization