Mixed Precision DNNs: All you need is a good parametrization

Abstract

Efficient deep neural network (DNN) inference on mobile or embedded devices typically involves quantization of the network parameters and activations. In particular, mixed precision networks achieve better performance than networks with homogeneous bitwidth for the same size constraint. Since choosing the optimal bitwidths is not straight forward, training methods, which can learn them, are desirable. Differentiable quantization with straight-through gradients allows to learn the quantizer's parameters using gradient methods. We show that a suited parametrization of the quantizer is the key to achieve a stable training and a good final performance. Specifically, we propose to parametrize the quantizer with the step size and dynamic range. The bitwidth can then be inferred from them. Other parametrizations, which explicitly use the bitwidth, consistently perform worse. We confirm our findings with experiments on CIFAR-10 and ImageNet and we obtain mixed precision DNNs with learned quantization parameters, achieving state-of-the-art performance.

International Conference on Learning Representations (ICLR) 2020; Source code at https://github.com/sony/ai-research-code

References in corpus (11)

• Categorical Reparameterization with Gumbel-Softmax
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Ternary Weight Networks
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• A Survey of FPGA-Based Neural Network Accelerator
• ProxQuant: Quantized Neural Networks via Proximal Operators
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• Iteratively Training Look-Up Tables for Network Quantization

Cited by in corpus (22)

• Bayesian Bits: Unifying Quantization and Pruning
• Up or Down? Adaptive Rounding for Post-Training Quantization
• Differentiable Model Compression via Pseudo Quantization Noise
• LSQ+: Improving low-bit quantization through learnable offsets and better initialization
• Iteratively Training Look-Up Tables for Network Quantization
• A White Paper on Neural Network Quantization
• Real-Time Video Super-Resolution on Smartphones with Deep Learning, Mobile AI 2021 Challenge: Report
• Sharpness-aware Quantization for Deep Neural Networks
• Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution
• Learnable Companding Quantization for Accurate Low-bit Neural Networks
• MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantization
• Confounding Tradeoffs for Neural Network Quantization
• WaveQ: Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Adaptive Regularization
• GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networks
• Semi-Relaxed Quantization with DropBits: Training Low-Bit Neural Networks via Bit-wise Regularization
• Filter Pre-Pruning for Improved Fine-tuning of Quantized Deep Neural Networks
• RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions
• Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimization
• Quantized Neural Networks via {-1, +1} Encoding Decomposition and Acceleration
• HPTQ: Hardware-Friendly Post Training Quantization
• Bayesian Sparsification Methods for Deep Complex-valued Networks
• QuantNet: Learning to Quantize by Learning within Fully Differentiable Framework