Knowledge Distillation as Semiparametric Inference
arXiv:2104.09732
Abstract
A popular approach to model compression is to train an inexpensive student model to mimic the class probabilities of a highly accurate but cumbersome teacher model. Surprisingly, this two-step knowledge distillation process often leads to higher accuracy than training the student directly on labeled data. To explain and enhance this phenomenon, we cast knowledge distillation as a semiparametric inference problem with the optimal student model as the target, the unknown Bayes class probabilities as nuisance, and the teacher probabilities as a plug-in nuisance estimate. By adapting modern semiparametric tools, we derive new guarantees for the prediction error of standard distillation and develop two enhancements -- cross-fitting and loss correction -- to mitigate the impact of teacher overfitting and underfitting on student performance. We validate our findings empirically on both tabular and image data and observe consistent improvements from our knowledge distillation enhancements.
References in corpus (15)
- Distilling the Knowledge in a Neural Network
- Knowledge Distillation: A Survey
- Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
- Empirical Bernstein Bounds and Sample Variance Penalization
- Data-Free Knowledge Distillation for Deep Neural Networks
- Towards Understanding Knowledge Distillation
- Understanding and Improving Knowledge Distillation
- And the Bit Goes Down: Revisiting the Quantization of Neural Networks
- Ensemble Distillation for Neural Machine Translation
- De-Biased Machine Learning of Global and Local Parameters Using Regularized Riesz Representers
- Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification
- Why distillation helps: a statistical perspective
- Teacher-Student Compression with Generative Adversarial Networks
- MLBench: How Good Are Machine Learning Clouds for Binary Classification Tasks on Structured Data?
- Regularized Orthogonal Machine Learning for Nonlinear Semiparametric Models