Up or Down? Adaptive Rounding for Post-Training Quantization

Abstract

When quantizing neural networks, assigning each floating-point weight to its nearest fixed-point value is the predominant approach. We find that, perhaps surprisingly, this is not the best we can do. In this paper, we propose AdaRound, a better weight-rounding mechanism for post-training quantization that adapts to the data and the task loss. AdaRound is fast, does not require fine-tuning of the network, and only uses a small amount of unlabelled data. We start by theoretically analyzing the rounding problem for a pre-trained neural network. By approximating the task loss with a Taylor series expansion, the rounding task is posed as a quadratic unconstrained binary optimization problem. We simplify this to a layer-wise local loss and propose to optimize this loss with a soft relaxation. AdaRound not only outperforms rounding-to-nearest by a significant margin but also establishes a new state-of-the-art for post-training quantization on several networks and tasks. Without fine-tuning, we can quantize the weights of Resnet18 and Resnet50 to 4 bits while staying within an accuracy loss of 1%.

Published as a conference paper at ICML 2020

References in corpus (15)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• A Survey on Methods and Theories of Quantized Neural Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
• Mixed Precision DNNs: All you need is a good parametrization
• QKD: Quantization-aware Knowledge Distillation
• Practical Gauss-Newton Optimisation for Deep Learning
• Fighting Quantization Bias With Bias
• ZeroQ: A Novel Zero Shot Quantization Framework

Cited by in corpus (5)

• Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
• An empirical study of LLaMA3 quantization: from LLMs to MLLMs
• Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
• Developing efficient transfer learning strategies for robust scene recognition in mobile robotics using pre-trained convolutional neural networks
• MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing