AdaBits: Neural Network Quantization with Adaptive Bit-Widths

Abstract

Deep neural networks with adaptive configurations have gained increasing attention due to the instant and flexible deployment of these models on platforms with different resource budgets. In this paper, we investigate a novel option to achieve this goal by enabling adaptive bit-widths of weights and activations in the model. We first examine the benefits and challenges of training quantized model with adaptive bit-widths, and then experiment with several approaches including direct adaptation, progressive training and joint training. We discover that joint training is able to produce comparable performance on the adaptive model as individual models. We further propose a new technique named Switchable Clipping Level (S-CL) to further improve quantized models at the lowest bit-width. With our proposed techniques applied on a bunch of models including MobileNet-V1/V2 and ResNet-50, we demonstrate that bit-width of weights and activations is a new option for adaptively executable deep neural networks, offering a distinct opportunity for improved accuracy-efficiency trade-off as well as instant adaptation according to the platform constraints in real-world applications.

arXiv commentCVPR 2020

References in corpus (20)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• A Survey of Deep Learning Techniques for Autonomous Driving
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Ternary Weight Networks
• Trained Ternary Quantization
• AutoAugment: Learning Augmentation Policies from Data
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• SNAS: Stochastic Neural Architecture Search
• Slimmable Neural Networks
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• WRPN: Wide Reduced-Precision Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Alternating Multi-bit Quantization for Recurrent Neural Networks
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• ProxQuant: Quantized Neural Networks via Proximal Operators
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• Towards Efficient Training for Neural Network Quantization

Cited by in corpus (5)

• MWQ: Multiscale Wavelet Quantized Neural Networks
• Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization
• Neural Architecture Search for Lightweight Non-Local Networks
• One Model for All Quantization: A Quantized Network Supporting Hot-Swap Bit-Width Adjustment
• InstantNet: Automated Generation and Deployment of Instantaneously Switchable-Precision Networks