Learnable Companding Quantization for Accurate Low-bit Neural Networks

Abstract

Quantizing deep neural networks is an effective method for reducing memory consumption and improving inference speed, and is thus useful for implementation in resource-constrained devices. However, it is still hard for extremely low-bit models to achieve accuracy comparable with that of full-precision models. To address this issue, we propose learnable companding quantization (LCQ) as a novel non-uniform quantization method for 2-, 3-, and 4-bit models. LCQ jointly optimizes model weights and learnable companding functions that can flexibly and non-uniformly control the quantization levels of weights and activations. We also present a new weight normalization technique that allows more stable training for quantization. Experimental results show that LCQ outperforms conventional state-of-the-art methods and narrows the gap between quantized and full-precision models for image classification and object detection tasks. Notably, the 2-bit ResNet-50 model on ImageNet achieves top-1 accuracy of 75.1% and reduces the gap to 1.7%, allowing LCQ to further exploit the potential of non-uniform quantization.

arXiv commentAccepted at CVPR 2021

References in corpus (7)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Convolutional Neural Networks using Logarithmic Data Representation
• Mixed Precision DNNs: All you need is a good parametrization
• Deep Neural Network Compression with Single and Multiple Level Quantization

Cited by in corpus (2)

• Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
• MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantization