Convolutional Neural Networks Quantization with Attention

Abstract

It has been proven that, compared to using 32-bit floating-point numbers in the training phase, Deep Convolutional Neural Networks (DCNNs) can operate with low precision during inference, thereby saving memory space and power consumption. However, quantizing networks is always accompanied by an accuracy decrease. Here, we propose a method, double-stage Squeeze-and-Threshold (double-stage ST). It uses the attention mechanism to quantize networks and achieve state-of-art results. Using our method, the 3-bit model can achieve accuracy that exceeds the accuracy of the full-precision baseline model. The proposed double-stage ST activation quantization is easy to apply: inserting it before the convolution.

arXiv commentPreprint of an article published in International Journal of Neural Systems, [10.1142/S0129065722500514] \c{opyright} [copyright World Scientific Publishing Company] [https://www.worldscientific.com/doi/10.1142/S0129065722500514]

References in corpus (7)

• TensorFlow: A system for large-scale machine learning
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• XNOR-Net++: Improved Binary Neural Networks
• NICE: Noise Injection and Clamping Estimation for Neural Network Quantization
• How Do Adam and Training Strategies Help BNNs Optimization?
• Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution