Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer

Abstract

Quantizing weights and activations of deep neural networks is essential for deploying them in resource-constrained devices, or cloud platforms for at-scale services. While binarization is a special case of quantization, this extreme case often leads to several training difficulties, and necessitates specialized models and training methods. As a result, recent quantization methods do not provide binarization, thus losing the most resource-efficient option, and quantized and binarized networks have been distinct research areas. We examine binarization difficulties in a quantization framework and find that all we need to enable the binary training are a symmetric quantizer, good initialization, and careful hyperparameter selection. These techniques also lead to substantial improvements in multi-bit quantization. We demonstrate our unified quantization framework, denoted as UniQ, on the ImageNet dataset with various architectures such as ResNet-18,-34 and MobileNetV2. For multi-bit quantization, UniQ outperforms existing methods to achieve the state-of-the-art accuracy. In binarization, the achieved accuracy is comparable to existing state-of-the-art methods even without modifying the original architectures.

References in corpus (15)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Binarized Neural Networks
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• Model compression via distillation and quantization
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• XNOR Neural Engine: a Hardware Accelerator IP for 21.6 fJ/op Binary Neural Network Inference
• XNOR-Net++: Improved Binary Neural Networks
• Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
• QKD: Quantization-aware Knowledge Distillation
• BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations
• Binarized Neural Networks on the ImageNet Classification Task
• Distillation Guided Residual Learning for Binary Convolutional Neural Networks
• AQD: Towards Accurate Fully-Quantized Object Detection