Learned Step Size Quantization

Abstract

Deep networks run with low precision operations at inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as precision decreases. Here, we present a method for training such networks, Learned Step Size Quantization, that achieves the highest accuracy to date on the ImageNet dataset when using models, from a variety of architectures, with weights and activations quantized to 2-, 3- or 4-bits of precision, and that can train 3-bit models that reach full precision baseline accuracy. Our approach builds upon existing methods for learning weights in quantized networks by improving how the quantizer itself is configured. Specifically, we introduce a novel means to estimate and scale the task loss gradient at each weight and activation layer's quantizer step size, such that it can be learned in conjunction with other network parameters. This approach works using different levels of precision as needed for a given system and requires only a simple modification of existing training code.

arXiv commentInternational Conference on Learning Representations (2020)

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Binarized Neural Networks
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• Ternary Weight Networks
• Trained Ternary Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Large Batch Training of Convolutional Networks
• Model compression via distillation and quantization
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Resiliency of Deep Neural Networks under Quantization
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)
• NICE: Noise Injection and Clamping Estimation for Neural Network Quantization

Cited by in corpus (41)

• Quantization and Deployment of Deep Neural Networks on Microcontrollers
• Real-Time Decoding for Fault-Tolerant Quantum Computing: Progress, Challenges and Outlook
• Resource-efficient Deep Neural Networks for Automotive Radar Interference Mitigation
• Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
• QKD: Quantization-aware Knowledge Distillation
• BinaryBERT: Pushing the Limit of BERT Quantization
• Robust Quantization: One Model to Rule Them All
• Towards Efficient Training for Neural Network Quantization
• Binary Neural Networks for Memory-Efficient and Effective Visual Place Recognition in Changing Environments
• Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers
• SamurAI: A Versatile IoT Node With Event-Driven Wake-Up and Embedded ML Acceleration
• Quantune: Post-training Quantization of Convolutional Neural Networks using Extreme Gradient Boosting for Fast Deployment
• Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
• Towards Unified INT8 Training for Convolutional Neural Network
• Flexible and Fully Quantized Ultra-Lightweight TinyissimoYOLO for Ultra-Low-Power Edge Systems
• LSQ+: Improving low-bit quantization through learnable offsets and better initialization
• N3H-Core: Neuron-designed Neural Network Accelerator via FPGA-based Heterogeneous Computing Cores
• Iteratively Training Look-Up Tables for Network Quantization
• I-BERT: Integer-only BERT Quantization
• BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization
• BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation
• CPT: Efficient Deep Neural Network Training via Cyclic Precision
• Towards Real-Time Neural Video Codec for Cross-Platform Application Using Calibration Information
• LegoNet: Memory Footprint Reduction Through Block Weight Clustering
• A 1Mb mixed-precision quantized encoder for image classification and patch-based compression
• Selective Focus: Investigating Semantics Sensitivity in Post-training Quantization for Lane Detection
• AUSN: Approximately Uniform Quantization by Adaptively Superimposing Non-uniform Distribution for Deep Neural Networks
• MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantization
• A High-Performance Adaptive Quantization Approach for Edge CNN Applications
• BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization
• Some Remarks on Replicated Simulated Annealing
• Histogram-Equalized Quantization for logic-gated Residual Neural Networks
• Fully Quantized Image Super-Resolution Networks
• Pruning Ternary Quantization
• MOGNET: A Mux-residual quantized Network leveraging Online-Generated weights
• HPTQ: Hardware-Friendly Post Training Quantization
• DNN Quantization with Attention
• DecisiveNets: Training Deep Associative Memories to Solve Complex Machine Learning Problems
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training