VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
arXiv:2102.04503
Abstract
Quantization enables efficient acceleration of deep neural networks by reducing model memory footprint and exploiting low-cost integer math hardware units. Quantization maps floating-point weights and activations in a trained model to low-bitwidth integer values using scale factors. Excessive quantization, reducing precision too aggressively, results in accuracy degradation. When scale factors are shared at a coarse granularity across many dimensions of each tensor, effective precision of individual elements within the tensor are limited. To reduce quantization-related accuracy loss, we propose using a separate scale factor for each small vector of (16-64) elements within a single dimension of a tensor. To achieve an efficient hardware implementation, the per-vector scale factors can be implemented with low-bitwidth integers when calibrated using a two-level quantization scheme. We find that per-vector scaling consistently achieves better inference accuracy at low precision compared to conventional scaling techniques for popular neural networks without requiring retraining. We also modify a deep learning accelerator hardware design to study the area and energy overheads of per-vector scaling support. Our evaluation demonstrates that per-vector scaled quantization with 4-bit weights and activations achieves 37% area saving and 24% energy saving while maintaining over 75% accuracy for ResNet50 on ImageNet. 4-bit weights and 8-bit activations achieve near-full-precision accuracy for both BERT-base and BERT-large on SQuAD while reducing area by 26% compared to an 8-bit baseline.
References in corpus (13)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Compressing Deep Convolutional Networks using Vector Quantization
- Quantizing deep convolutional networks for efficient inference: A whitepaper
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- Convolutional Neural Networks using Logarithmic Data Representation
- Training deep neural networks with low precision multiplications
- Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
- Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
- Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
- Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model
- Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
- Quantization for Rapid Deployment of Deep Neural Networks