Data Efficient Stagewise Knowledge Distillation
arXiv:1911.06786
Abstract
Despite the success of Deep Learning (DL), the deployment of modern DL models requiring large computational power poses a significant problem for resource-constrained systems. This necessitates building compact networks that reduce computations while preserving performance. Traditional Knowledge Distillation (KD) methods that transfer knowledge from teacher to student (a) use a single-stage and (b) require the whole data set while distilling the knowledge to the student. In this work, we propose a new method called Stagewise Knowledge Distillation (SKD) which builds on traditional KD methods by progressive stagewise training to leverage the knowledge gained from the teacher, resulting in data-efficient distillation process. We evaluate our method on classification and semantic segmentation tasks. We show, across the tested tasks, significant performance gains even with a fraction of the data used in distillation, without compromising on the metric. We also compare our method with existing KD techniques and show that SKD outperforms them. Moreover, our method can be viewed as a generalized model compression technique that complements other model compression methods such as quantization or pruning.
arXiv comment15 pages, 1 figure, 6 tables and 1 algorithm
References in corpus (16)
- Distilling the Knowledge in a Neural Network
- Densely Connected Convolutional Networks
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Do Deep Nets Really Need to be Deep?
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- fastai: A Layered API for Deep Learning
- Compressing Deep Convolutional Networks using Vector Quantization
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- Model compression via distillation and quantization
- Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units
- Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
- N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
- Improving Fast Segmentation With Teacher-student Learning
- Quantized Convolutional Neural Networks for Mobile Devices
- MEAL: Multi-Model Ensemble via Adversarial Learning