QKD: Quantization-aware Knowledge Distillation

Abstract

Quantization and Knowledge distillation (KD) methods are widely used to reduce memory and power consumption of deep neural networks (DNNs), especially for resource-constrained edge devices. Although their combination is quite promising to meet these requirements, it may not work as desired. It is mainly because the regularization effect of KD further diminishes the already reduced representation power of a quantized model. To address this short-coming, we propose Quantization-aware Knowledge Distillation (QKD) wherein quantization and KD are care-fully coordinated in three phases. First, Self-studying (SS) phase fine-tunes a quantized low-precision student network without KD to obtain a good initialization. Second, Co-studying (CS) phase tries to train a teacher to make it more quantizaion-friendly and powerful than a fixed teacher. Finally, Tutoring (TU) phase transfers knowledge from the trained teacher to the student. We extensively evaluate our method on ImageNet and CIFAR-10/100 datasets and show an ablation study on networks with both standard and depthwise-separable convolutions. The proposed QKD outperformed existing state-of-the-art methods (e.g., 1.3% improvement on ResNet-18 with W4A4, 2.6% on MobileNetV2 with W4A4). Additionally, QKD could recover the full-precision accuracy at as low as W3A3 quantization on ResNet and W6A6 quantization on MobilenetV2.

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• FitNets: Hints for Thin Deep Nets
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Binarized Neural Networks
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Ternary Weight Networks
• Trained Ternary Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• Learned Step Size Quantization
• Model compression via distillation and quantization
• WRPN: Wide Reduced-Precision Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Resiliency of Deep Neural Networks under Quantization
• Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model
• Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
• Feature Fusion for Online Mutual Knowledge Distillation
• A Comprehensive Overhaul of Feature Distillation

Cited by in corpus (13)

• Knowledge Distillation: A Survey
• Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
• Up or Down? Adaptive Rounding for Post-Training Quantization
• Feature Fusion for Online Mutual Knowledge Distillation
• LSQ+: Improving low-bit quantization through learnable offsets and better initialization
• BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• TernaryBERT: Distillation-aware Ultra-low Bit BERT
• ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
• Dynamic Network Quantization for Efficient Video Inference
• Arch-Net: Model Distillation for Architecture Agnostic Model Deployment
• Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths
• Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer