Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming

Abstract

Lately, post-training quantization methods have gained considerable attention, as they are simple to use, and require only a small unlabeled calibration set. This small dataset cannot be used to fine-tune the model without significant over-fitting. Instead, these methods only use the calibration set to set the activations' dynamic ranges. However, such methods always resulted in significant accuracy degradation, when used below 8-bits (except on small datasets). Here we aim to break the 8-bit barrier. To this end, we minimize the quantization errors of each layer separately by optimizing its parameters over the calibration set. We empirically demonstrate that this approach is: (1) much less susceptible to over-fitting than the standard fine-tuning approaches, and can be used even on a very small calibration set; and (2) more powerful than previous methods, which only set the activations' dynamic ranges. Furthermore, we demonstrate how to optimally allocate the bit-widths for each layer, while constraining accuracy degradation or model compression by proposing a novel integer programming formulation. Finally, we suggest model global statistics tuning, to correct biases introduced during quantization. Together, these methods yield state-of-the-art results for both vision and text models. For instance, on ResNet50, we obtain less than 1\% accuracy degradation --- with 4-bit weights and activations in all layers, but the smallest two. We open-sourced our code.

References in corpus (13)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Fixed Point Quantization of Deep Convolutional Networks
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Up or Down? Adaptive Rounding for Post-Training Quantization
• QKD: Quantization-aware Knowledge Distillation
• Fighting Quantization Bias With Bias
• ZeroQ: A Novel Zero Shot Quantization Framework
• Knapsack Pruning with Inner Distillation

Cited by in corpus (14)

• BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
• Rethinking "Batch" in BatchNorm
• Towards Efficient Post-training Quantization of Pre-trained Language Models
• HAWQV3: Dyadic Neural Network Quantization
• Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks
• VLCQ: Post-training quantization for deep neural networks using variable length coding
• Selective Focus: Investigating Semantics Sensitivity in Post-training Quantization for Lane Detection
• Fine-grained Data Distribution Alignment for Post-Training Quantization
• Confounding Tradeoffs for Neural Network Quantization
• Post-Training BatchNorm Recalibration
• Exploring Neural Networks Quantization via Layer-Wise Quantization Analysis
• Arch-Net: Model Distillation for Architecture Agnostic Model Deployment
• PTQ-SL: Exploring the Sub-layerwise Post-training Quantization
• HPTQ: Hardware-Friendly Post Training Quantization