Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks

Abstract

We propose a method of training quantization thresholds (TQT) for uniform symmetric quantizers using standard backpropagation and gradient descent. Contrary to prior work, we show that a careful analysis of the straight-through estimator for threshold gradients allows for a natural range-precision trade-off leading to better optima. Our quantizers are constrained to use power-of-2 scale-factors and per-tensor scaling of weights and activations to make it amenable for hardware implementations. We present analytical support for the general robustness of our methods and empirically validate them on various CNNs for ImageNet classification. We are able to achieve near-floating-point accuracy on traditionally difficult networks such as MobileNets with less than 5 epochs of quantized (8-bit) retraining. Finally, we present Graffitist, a framework that enables automatic quantization of TensorFlow graphs for TQT (available at https://github.com/Xilinx/graffitist ).

arXiv commentLink to Conference (Oral & Poster) Schedule - https://mlsys.org/Conferences/2020/ScheduleMultitrack?event=1431

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• TensorFlow: A system for large-scale machine learning
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Going Deeper with Convolutions
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Ternary Weight Networks
• Trained Ternary Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• YOLO9000: Better, Faster, Stronger
• Learned Step Size Quantization
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• WRPN: Wide Reduced-Precision Networks
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks
• NICE: Noise Injection and Clamping Estimation for Neural Network Quantization

Cited by in corpus (30)

• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Bringing AI To Edge: From Deep Learning's Perspective
• Bayesian Bits: Unifying Quantization and Pruning
• Resource-efficient Deep Neural Networks for Automotive Radar Interference Mitigation
• Up or Down? Adaptive Rounding for Post-Training Quantization
• QKD: Quantization-aware Knowledge Distillation
• EasyQuant: Post-training Quantization via Scale Optimization
• Low-Memory Neural Network Training: A Technical Report
• Towards Efficient Training for Neural Network Quantization
• Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers
• Pareto-Optimal Quantized ResNet Is Mostly 4-bit
• Differentiable Model Compression via Pseudo Quantization Noise
• LSQ+: Improving low-bit quantization through learnable offsets and better initialization
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• Fast and Accurate Single-Image Depth Estimation on Mobile Devices, Mobile AI 2021 Challenge: Report
• Real-Time Video Super-Resolution on Smartphones with Deep Learning, Mobile AI 2021 Challenge: Report
• AnalogNets: ML-HW Co-Design of Noise-robust TinyML Models and Always-On Analog Compute-in-Memory Accelerator
• Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization
• Neural gradients are near-lognormal: improved quantized and sparse training
• HYPER-SNN: Towards Energy-efficient Quantized Deep Spiking Neural Networks for Hyperspectral Image Classification
• Some Remarks on Replicated Simulated Annealing
• Real-Time Quantized Image Super-Resolution on Mobile NPUs, Mobile AI 2021 Challenge: Report
• All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networks
• Dual Precision Deep Neural Network
• Accelerating Neural Network Inference by Overflow Aware Quantization
• Fast Camera Image Denoising on Mobile GPUs with Deep Learning, Mobile AI 2021 Challenge: Report
• HPTQ: Hardware-Friendly Post Training Quantization
• Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer
• Semi-Relaxed Quantization with DropBits: Training Low-Bit Neural Networks via Bit-wise Regularization
• Compressed Object Detection