Does Knowledge Distillation Really Work?
arXiv:2106.05945
Abstract
Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks. We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood: there often remains a surprisingly large discrepancy between the predictive distributions of the teacher and the student, even in cases when the student has the capacity to perfectly match the teacher. We identify difficulties in optimization as a key reason for why the student is unable to match the teacher. We also show how the details of the dataset used for distillation play a role in how closely the student matches the teacher -- and that more closely matching the teacher paradoxically does not always lead to better student generalization.
NeurIPS 2021. Code available at https://github.com/samuelstanton/gnosis
References in corpus (4)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- On Calibration of Modern Neural Networks
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
Cited by in corpus (5)
- Deep Neural Compression Via Concurrent Pruning and Self-Distillation
- A Fast Knowledge Distillation Framework for Visual Recognition
- Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation
- AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family
- A Survey on Green Deep Learning