Improving Neural Network Quantization without Retraining using Outlier Channel Splitting

Abstract

Quantization can improve the execution latency and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs, while this work examines the less-studied topic of quantizing a floating-point model without (re)training. DNN weights and activations follow a bell-shaped distribution post-training, while practical hardware uses a linear quantization grid. This leads to challenges in dealing with outliers in the distribution. Prior work has addressed this by clipping the outliers or using specialized hardware. In this work, we propose outlier channel splitting (OCS), which duplicates channels containing outliers, then halves the channel values. The network remains functionally identical, but affected outliers are moved toward the center of the distribution. OCS requires no additional training and works on commodity hardware. Experimental evaluation on ImageNet classification and language modeling shows that OCS can outperform state-of-the-art clipping techniques with only minor overhead.

10 pages; update to ICML camera-ready version

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Recurrent Neural Network Regularization
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Pointer Sentinel Mixture Models
• Resiliency of Deep Neural Networks under Quantization
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)
• Quantizing Convolutional Neural Networks for Low-Power High-Throughput Inference Engines

Cited by in corpus (42)

• Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
• Applications and Techniques for Fast Machine Learning in Science
• Autoencoders on FPGAs for real-time, unsupervised new physics detection at 40 MHz at the Large Hadron Collider
• Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
• Up or Down? Adaptive Rounding for Post-Training Quantization
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• BinaryBERT: Pushing the Limit of BERT Quantization
• Robust Quantization: One Model to Rule Them All
• Lightweight Compression of Intermediate Neural Network Features for Collaborative Intelligence
• DSConv: Efficient Convolution Operator
• Controlling Information Capacity of Binary Neural Network
• Pareto-Optimal Quantized ResNet Is Mostly 4-bit
• Knowledge Distillation Methods for Efficient Unsupervised Adaptation Across Multiple Domains
• Quantune: Post-training Quantization of Convolutional Neural Networks using Extreme Gradient Boosting for Fast Deployment
• ZeroQ: A Novel Zero Shot Quantization Framework
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Post-Training Piecewise Linear Quantization for Deep Neural Networks
• Generative Low-bitwidth Data Free Quantization
• Automated Design Space Exploration for optimised Deployment of DNN on Arm Cortex-A CPUs
• Sparse Weight Activation Training
• RPR: Random Partition Relaxation for Training; Binary and Ternary Weight Neural Networks
• A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statistics
• Diversifying Sample Generation for Accurate Data-Free Quantization
• Diverse Sample Generation: Pushing the Limit of Generative Data-free Quantization
• Post-training Quantization with Multiple Points: Mixed Precision without Mixed Precision
• Quantized Convolutional Neural Networks Through the Lens of Partial Differential Equations
• OverQ: Opportunistic Outlier Quantization for Neural Network Accelerators
• Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomes
• Weight Equalizing Shift Scaler-Coupled Post-training Quantization
• GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networks
• Q-Rater: Non-Convex Optimization for Post-Training Uniform Quantization
• An Inter-Layer Weight Prediction and Quantization for Deep Neural Networks based on a Smoothly Varying Weight Hypothesis
• Arch-Net: Model Distillation for Architecture Agnostic Model Deployment
• HERO: Hessian-Enhanced Robust Optimization for Unifying and Improving Generalization and Quantization Performance
• Search-Optimized Quantization in Biomedical Ontology Alignment
• Is In-Domain Data Really Needed? A Pilot Study on Cross-Domain Calibration for Network Quantization
• HPTQ: Hardware-Friendly Post Training Quantization
• Auto-Split: A General Framework of Collaborative Edge-Cloud AI
• Hybrid and Non-Uniform quantization methods using retro synthesis data for efficient inference
• From Quantized DNNs to Quantizable DNNs
• Neural Network Activation Quantization with Bitwise Information Bottlenecks
• Distilling Image Classifiers in Object Detectors