Precision Highway for Ultra Low-Precision Quantization

Abstract

Neural network quantization has an inherent problem called accumulated quantization error, which is the key obstacle towards ultra-low precision, e.g., 2- or 3-bit precision. To resolve this problem, we propose precision highway, which forms an end-to-end high-precision information flow while performing the ultra low-precision computation. First, we describe how the precision highway reduce the accumulated quantization error in both convolutional and recurrent neural networks. We also provide the quantitative analysis of the benefit of precision highway and evaluate the overhead on the state-of-the-art hardware accelerator. In the experiments, our proposed method outperforms the best existing quantization methods while offering 3-bit weight/activation quantization with no accuracy loss and 2-bit quantization with a 2.45 % top-1 accuracy loss in ResNet-50. We also report that the proposed method significantly outperforms the existing method in the 2-bit quantization of an LSTM for language modeling.

References in corpus (13)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Recurrent Neural Network Regularization
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Trained Ternary Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Visualizing the Loss Landscape of Neural Nets
• WRPN: Wide Reduced-Precision Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Effective Quantization Methods for Recurrent Neural Networks
• Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)
• Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks
• Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm
• High performance ultra-low-precision convolutions on mobile devices

Cited by in corpus (4)

• A Learning Framework for n-bit Quantized Neural Networks toward FPGAs
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
• Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks
• Confounding Tradeoffs for Neural Network Quantization