Knowledge distillation for optimization of quantized deep neural networks
arXiv:1909.01688
Abstract
Knowledge distillation (KD) is a very popular method for model size reduction. Recently, the technique is exploited for quantized deep neural networks (QDNNs) training as a way to restore the performance sacrificed by word-length reduction. KD, however, employs additional hyper-parameters, such as temperature, coefficient, and the size of teacher network for QDNN training. We analyze the effect of these hyper-parameters for QDNN optimization with KD. We find that these hyper-parameters are inter-related, and also introduce a simple and effective technique that reduces \textit{coefficient} during training. With KD employing the proposed hyper-parameters, we achieve the test accuracy of 92.7% and 67.0% on Resnet20 with 2-bit ternary weights for CIFAR-10 and CIFAR-100 data sets, respectively.
References in corpus (15)
- Distilling the Knowledge in a Neural Network
- FitNets: Hints for Thin Deep Nets
- Wide Residual Networks
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Deep Learning with Limited Numerical Precision
- Resiliency of Deep Neural Networks under Quantization
- Recurrent Neural Network Training with Dark Knowledge Transfer
- Deep Learning with Low Precision by Half-wave Gaussian Quantization
- Effective Quantization Methods for Recurrent Neural Networks
- Recurrent Neural Networks With Limited Numerical Precision
- LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks
- Low Precision RNNs: Quantizing RNNs Without Losing Accuracy
- Knowledge distillation using unlabeled mismatched images