Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
arXiv:2004.09602
Abstract
Quantization techniques can reduce the size of Deep Neural Networks and improve inference latency and throughput by taking advantage of high throughput integer instructions. In this paper we review the mathematical aspects of quantization parameters and evaluate their choices on a wide range of neural network models for different application domains, including vision, speech, and language. We focus on quantization techniques that are amenable to acceleration by processors with high-throughput integer math pipelines. We also present a workflow for 8-bit quantization that is able to maintain accuracy within 1% of the floating-point baseline on all networks studied, including models that are more difficult to quantize, such as MobileNets and BERT-large.
20 pages, 7 figures
References in corpus (11)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- cuDNN: Efficient Primitives for Deep Learning
- Searching for Activation Functions
- Trained Ternary Quantization
- Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
- Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
- Ternary Neural Networks with Fine-Grained Quantization
- Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model
- Neural Network Distiller: A Python Package For DNN Compression Research
- Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers