Follow Your Path: a Progressive Method for Knowledge Distillation
arXiv:2107.09305
Abstract
Deep neural networks often have a huge number of parameters, which posts challenges in deployment in application scenarios with limited memory and computation capacity. Knowledge distillation is one approach to derive compact models from bigger ones. However, it has been observed that a converged heavy teacher model is strongly constrained for learning a compact student network and could make the optimization subject to poor local optima. In this paper, we propose ProKT, a new model-agnostic method by projecting the supervision signals of a teacher model into the student's parameter space. Such projection is implemented by decomposing the training objective into local intermediate targets with an approximate mirror descent technique. The proposed method could be less sensitive with the quirks during optimization which could result in a better local optimum. Experiments on both image and text datasets show that our proposed ProKT consistently achieves superior performance compared to other existing knowledge distillation methods.
arXiv commentAccepted by ECML-PKDD 2021
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Wide Residual Networks
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
- Model compression via distillation and quantization
- Towards Understanding Knowledge Distillation
- Contrastive Representation Distillation
- AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods
- CoT: Cooperative Training for Generative Modeling of Discrete Data