Blended Coarse Gradient Descent for Full Quantization of Deep Neural Networks

Abstract

Quantized deep neural networks (QDNNs) are attractive due to their much lower memory storage and faster inference speed than their regular full precision counterparts. To maintain the same performance level especially at low bit-widths, QDNNs must be retrained. Their training involves piecewise constant activation functions and discrete weights, hence mathematical challenges arise. We introduce the notion of coarse gradient and propose the blended coarse gradient descent (BCGD) algorithm, for training fully quantized neural networks. Coarse gradient is generally not a gradient of any function but an artificial ascent direction. The weight update of BCGD goes by coarse gradient correction of a weighted average of the full precision weights and their quantization (the so-called blending), which yields sufficient descent in the objective value and thus accelerates the training. Our experiments demonstrate that this simple blending technique is very effective for quantization at extremely low bit-width such as binarization. In full quantization of ResNet-18 for ImageNet classification task, BCGD gives 64.36\% top-1 accuracy with binary weights across all layers and 4-bit adaptive activation. If the weights in the first and last layers are kept in full precision, this number increases to 65.46\%. As theoretical justification, we show convergence analysis of coarse gradient descent for a two-linear-layer neural network model with Gaussian input data, and prove that the expected coarse gradient correlates positively with the underlying true gradient.

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Ternary Weight Networks
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
• Deep Neural Nets with Interpolating Function as Output Activation

Cited by in corpus (8)

• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• Elastic Significant Bit Quantization and Acceleration for Deep Neural Networks
• Median Binary-Connect Method and a Binary Convolutional Neural Nework for Word Recognition
• AutoQNN: An End-to-End Framework for Automatically Quantizing Neural Networks
• QuantNet: Learning to Quantize by Learning within Fully Differentiable Framework
• Semi-Relaxed Quantization with DropBits: Training Low-Bit Neural Networks via Bit-wise Regularization
• BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model
• DAC: Data-free Automatic Acceleration of Convolutional Networks