Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs

Abstract

Quantization is a technique for reducing deep neural networks (DNNs) training and inference times, which is crucial for training in resource constrained environments or applications where inference is time critical. State-of-the-art (SOTA) quantization approaches focus on post-training quantization, i.e., quantization of pre-trained DNNs for speeding up inference. While work on quantized training exists, most approaches require refinement in full precision (usually single precision) in the final training phase or enforce a global word length across the entire DNN. This leads to suboptimal assignments of bit-widths to layers and, consequently, suboptimal resource usage. In an attempt to overcome such limitations, we introduce AdaPT, a new fixed-point quantized sparsifying training strategy. AdaPT decides about precision switches between training epochs based on information theoretic conditions. The goal is to determine on a per-layer basis the lowest precision that causes no quantization-induced information loss while keeping the precision high enough such that future learning steps do not suffer from vanishing gradients. The benefits of the resulting fully quantized DNN are evaluated based on an analytical performance model which we develop. We illustrate that an average speedup of 1.27 compared to standard training in float32 with an average accuracy increase of 0.98% can be achieved for AlexNet/ResNet on CIFAR10/100 and we further demonstrate these AdaPT trained models achieve an average inference speedup of 2.33 with a model size reduction of 0.52.

arXiv comment14 pages, 8 figures, preprint

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• A Convergence Theory for Deep Learning via Over-Parameterization
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• UNIQ: Uniform Noise Injection for Non-Uniform Quantization of Neural Networks
• Training Quantized Nets: A Deeper Understanding
• Loss-aware Weight Quantization of Deep Networks
• LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networks
• HEMP: High-order Entropy Minimization for neural network comPression
• Multi-Precision Policy Enforced Training (MuPPET): A precision-switching strategy for quantised fixed-point training of CNNs
• TensorQuant - A Simulation Toolbox for Deep Neural Network Quantization

Cited by in corpus (1)

• Attacking Graph Neural Networks with Bit Flips: Weisfeiler and Lehman Go Indifferent