Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
arXiv:2111.14826
Abstract
The nonuniform quantization strategy for compressing neural networks usually achieves better performance than its counterpart, i.e., uniform strategy, due to its superior representational capacity. However, many nonuniform quantization methods overlook the complicated projection process in implementing the nonuniformly quantized weights/activations, which incurs non-negligible time and space overhead in hardware deployment. In this study, we propose Nonuniform-to-Uniform Quantization (N2UQ), a method that can maintain the strong representation ability of nonuniform methods while being hardware-friendly and efficient as the uniform quantization for model inference. We achieve this through learning the flexible in-equidistant input thresholds to better fit the underlying distribution while quantizing these real-valued inputs into equidistant output levels. To train the quantized network with learnable input thresholds, we introduce a generalized straight-through estimator (G-STE) for intractable backward derivative calculation w.r.t. threshold parameters. Additionally, we consider entropy preserving regularization to further reduce information loss in weight quantization. Even under this adverse constraint of imposing uniformly quantized weights and activations, our N2UQ outperforms state-of-the-art nonuniform quantization methods by 0.5~1.7 on ImageNet, demonstrating the contribution of N2UQ design. Code and models are available at: https://github.com/liuzechun/Nonuniform-to-Uniform-Quantization.
arXiv commentCVPR 2022
References in corpus (25)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Deep Residual Learning for Image Recognition
- MobileNetV2: Inverted Residuals and Linear Bottlenecks
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- Convolutional Neural Networks using Logarithmic Data Representation
- ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
- Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
- Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
- MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?
- Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
- LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks
- Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm
- Towards Effective Low-bitwidth Convolutional Neural Networks
- Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
- MEAL: Multi-Model Ensemble via Adversarial Learning
- BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNs
- Learnable Companding Quantization for Accurate Low-bit Neural Networks
- Distance-aware Quantization