Towards Efficient Training for Neural Network Quantization

Abstract

Quantization reduces computation costs of neural networks but suffers from performance degeneration. Is this accuracy drop due to the reduced capacity, or inefficient training during the quantization procedure? After looking into the gradient propagation process of neural networks by viewing the weights and intermediate activations as random variables, we discover two critical rules for efficient training. Recent quantization approaches violates the two rules and results in degenerated convergence. To deal with this problem, we propose a simple yet effective technique, named scale-adjusted training (SAT), to comply with the discovered rules and facilitates efficient training. We also analyze the quantization error introduced in calculating the gradient in the popular parameterized clipping activation (PACT) technique. Through SAT together with gradient-calibrated PACT, quantized models obtain comparable or even better performance than their full-precision counterparts, achieving state-of-the-art accuracy with consistent improvement over previous quantization methods on a wide spectrum of models including MobileNet-V1/V2 and PreResNet-50.

23 pages, 8 figures

References in corpus (26)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Continuous control with deep reinforcement learning
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Ternary Weight Networks
• Trained Ternary Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Learned Step Size Quantization
• Slimmable Neural Networks
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• WRPN: Wide Reduced-Precision Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Fixup Initialization: Residual Learning Without Normalization
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• Ternary Neural Networks with Fine-Grained Quantization
• Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
• A Mean Field Theory of Batch Normalization
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
• AutoQ: Automated Kernel-Wise Neural Network Quantization
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations

Cited by in corpus (6)

• SamurAI: A Versatile IoT Node With Event-Driven Wake-Up and Embedded ML Acceleration
• HCM: Hardware-Aware Complexity Metric for Neural Network Architectures
• AdaBits: Neural Network Quantization with Adaptive Bit-Widths
• Teachers Do More Than Teach: Compressing Image-to-Image Models
• Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution
• Is In-Domain Data Really Needed? A Pilot Study on Cross-Domain Calibration for Network Quantization