DNN Quantization with Attention

Abstract

Low-bit quantization of network weights and activations can drastically reduce the memory footprint, complexity, energy consumption and latency of Deep Neural Networks (DNNs). However, low-bit quantization can also cause a considerable drop in accuracy, in particular when we apply it to complex learning tasks or lightweight DNN architectures. In this paper, we propose a training procedure that relaxes the low-bit quantization. We call this procedure \textit{DNN Quantization with Attention} (DQA). The relaxation is achieved by using a learnable linear combination of high, medium and low-bit quantizations. Our learning procedure converges step by step to a low-bit quantization using an attention mechanism with temperature scheduling. In experiments, our approach outperforms other low-bit quantization techniques on various object recognition benchmarks such as CIFAR10, CIFAR100 and ImageNet ILSVRC 2012, achieves almost the same accuracy as a full precision DNN, and considerably reduces the accuracy drop when quantizing lightweight DNN architectures.

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• Binarized Neural Networks
• Ternary Weight Networks
• Trained Ternary Quantization
• Pruning Filters for Efficient ConvNets
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Born Again Neural Networks
• Fractional Max-Pooling
• Learned Step Size Quantization
• Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)
• Iteratively Training Look-Up Tables for Network Quantization
• BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization
• LIT: Block-wise Intermediate Representation Training for Model Compression