Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation

Abstract

Quantization techniques can reduce the size of Deep Neural Networks and improve inference latency and throughput by taking advantage of high throughput integer instructions. In this paper we review the mathematical aspects of quantization parameters and evaluate their choices on a wide range of neural network models for different application domains, including vision, speech, and language. We focus on quantization techniques that are amenable to acceleration by processors with high-throughput integer math pipelines. We also present a workflow for 8-bit quantization that is able to maintain accuracy within 1% of the floating-point baseline on all networks studied, including models that are more difficult to quantize, such as MobileNets and BERT-large.

20 pages, 7 figures

References in corpus (29)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Gaussian Error Linear Units (GELUs)
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• cuDNN: Efficient Primitives for Deep Learning
• Mixed Precision Training
• Binarized Neural Networks
• Searching for Activation Functions
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Trained Ternary Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Q8BERT: Quantized 8Bit BERT
• Training deep neural networks with low precision multiplications
• Model compression via distillation and quantization
• Three Factors Influencing Minima in SGD
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Image Classification at Supercomputer Scale
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• Ternary Neural Networks with Fine-Grained Quantization
• Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model
• Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
• Neural Network Distiller: A Python Package For DNN Compression Research
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers
• Retraining-Based Iterative Weight Quantization for Deep Neural Networks

Cited by in corpus (28)

• Evaluating the Single-Shot MultiBox Detector and YOLO Deep Learning Models for the Detection of Tomatoes in a Greenhouse
• Review: Deep Learning in Electron Microscopy
• An Electro-Photonic System for Accelerating Deep Neural Networks
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• RAELLA: Reforming the Arithmetic for Efficient, Low-Resolution, and Low-Loss Analog PIM: No Retraining Required!
• Pareto-Optimal Quantized ResNet Is Mostly 4-bit
• Efficient Quantized Sparse Matrix Operations on Tensor Cores
• Evaluating Single Event Upsets in Deep Neural Networks for Semantic Segmentation: an embedded system perspective
• Quantune: Post-training Quantization of Convolutional Neural Networks using Extreme Gradient Boosting for Fast Deployment
• Degree-Quant: Quantization-Aware Training for Graph Neural Networks
• Accelerating Sparse Deep Neural Networks
• Mirage: An RNS-Based Photonic Accelerator for DNN Training
• VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
• Quantizing YOLOv7: A Comprehensive Study
• Shifting Capsule Networks from the Cloud to the Deep Edge
• VLCQ: Post-training quantization for deep neural networks using variable length coding
• Training Deep Neural Networks with Joint Quantization and Pruning of Weights and Activations
• Implementing Reinforcement Learning Datacenter Congestion Control in NVIDIA NICs
• Quantized neural network for complex hologram generation
• At-Scale Evaluation of Weight Clustering to Enable Energy-Efficient Object Detection
• Differentiable Joint Pruning and Quantization for Hardware Efficiency
• Exploring Neural Networks Quantization via Layer-Wise Quantization Analysis
• Search-Optimized Quantization in Biomedical Ontology Alignment
• 4-bit Quantization of LSTM-based Speech Recognition Models
• Empirical Evaluation of Deep Learning Model Compression Techniques on the WaveNet Vocoder
• Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale
• HPTQ: Hardware-Friendly Post Training Quantization