Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)

Abstract

Deep learning algorithms achieve high classification accuracy at the expense of significant computation cost. In order to reduce this cost, several quantization schemes have gained attention recently with some focusing on weight quantization, and others focusing on quantizing activations. This paper proposes novel techniques that target weight and activation quantizations separately resulting in an overall quantized neural network (QNN). The activation quantization technique, PArameterized Clipping acTivation (PACT), uses an activation clipping parameter $\alpha$ that is optimized during training to find the right quantization scale. The weight quantization scheme, statistics-aware weight binning (SAWB), finds the optimal scaling factor that minimizes the quantization error based on the statistical characteristics of the distribution of weights without the need for an exhaustive search. The combination of PACT and SAWB results in a 2-bit QNN that achieves state-of-the-art classification accuracy (comparable to full precision networks) across a range of popular models and datasets.

arXiv commentarXiv admin note: substantial text overlap with arXiv:1805.06085

References in corpus (6)

• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Wide Residual Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Ternary Weight Networks
• Trained Ternary Quantization
• Training wide residual networks for deployment using a single bit for each weight

Cited by in corpus (25)

• Learned Step Size Quantization
• Quantization and Deployment of Deep Neural Networks on Microcontrollers
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Mixed-precision deep learning based on computational memory
• Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
• A Learning Framework for n-bit Quantized Neural Networks toward FPGAs
• Feature Map Transform Coding for Energy-Efficient CNN Inference
• Training large-scale ANNs on simulated resistive crossbar arrays
• On Periodic Functions as Regularizers for Quantization of Neural Networks
• Sparse Weight Activation Training
• Precision Highway for Ultra Low-Precision Quantization
• CAT: Compression-Aware Training for bandwidth reduction
• Channel-wise Hessian Aware trace-Weighted Quantization of Neural Networks
• SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks
• Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks
• Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution
• WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
• Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks
• Neural gradients are near-lognormal: improved quantized and sparse training
• Exploring Neural Networks Quantization via Layer-Wise Quantization Analysis
• Tuning Algorithms and Generators for Efficient Edge Inference
• One Weight Bitwidth to Rule Them All
• Pruning Ternary Quantization
• DNN Quantization with Attention
• 4-bit Quantization of LSTM-based Speech Recognition Models