A Statistical Framework for Low-bitwidth Training of Deep Neural Networks

Abstract

Fully quantized training (FQT), which uses low-bitwidth hardware by quantizing the activations, weights, and gradients of a neural network model, is a promising approach to accelerate the training of deep neural networks. One major challenge with FQT is the lack of theoretical understanding, in particular of how gradient quantization impacts convergence properties. In this paper, we address this problem by presenting a statistical framework for analyzing FQT algorithms. We view the quantized gradient of FQT as a stochastic estimator of its full precision counterpart, a procedure known as quantization-aware training (QAT). We show that the FQT gradient is an unbiased estimator of the QAT gradient, and we discuss the impact of gradient quantization on its variance. Inspired by these theoretical results, we develop two novel gradient quantizers, and we show that these have smaller variance than the existing per-tensor quantizer. For training ResNet-50 on ImageNet, our 5-bit block Householder quantizer achieves only 0.5% validation accuracy loss relative to QAT, comparable to the existing INT8 baseline.

24 pages

References in corpus (10)

• Language Models are Few-Shot Learners
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Post-training 4-bit quantization of convolution networks for rapid-deployment
• HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• Training DNNs with Hybrid Block Floating Point
• Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks

Cited by in corpus (4)

• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks