Why distillation helps: a statistical perspective

Abstract

Knowledge distillation is a technique for improving the performance of a simple "student" model by replacing its one-hot training labels with a distribution over labels obtained from a complex "teacher" model. While this simple approach has proven widely effective, a basic question remains unresolved: why does distillation help? In this paper, we present a statistical perspective on distillation which addresses this question, and provides a novel connection to extreme multiclass retrieval techniques. Our core observation is that the teacher seeks to estimate the underlying (Bayes) class-probability function. Building on this, we establish a fundamental bias-variance tradeoff in the student's objective: this quantifies how approximate knowledge of these class-probabilities can significantly aid learning. Finally, we show how distillation complements existing negative mining techniques for extreme multiclass retrieval, and propose a unified objective which combines these ideas.

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• Do Deep Nets Really Need to be Deep?
• When Does Label Smoothing Help?
• Empirical Bernstein Bounds and Sample Variance Penalization
• Towards Understanding Knowledge Distillation
• Recurrent Neural Network Training with Dark Knowledge Transfer
• Understanding and Improving Knowledge Distillation
• Zero-Shot Knowledge Distillation in Deep Networks
• Policy Distillation
• Stochastic Negative Mining for Learning with Large Output Spaces
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network

Cited by in corpus (9)

• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Self-Distillation Amplifies Regularization in Hilbert Space
• Knowledge Distillation as Semiparametric Inference
• Towards Model Agnostic Federated Learning Using Knowledge Distillation
• Churn Reduction via Distillation
• On Focal Loss for Class-Posterior Probability Estimation: A Theoretical Perspective
• When in Doubt, Summon the Titans: Efficient Inference with Large Models
• Distilling Double Descent
• Dual-Teacher++: Exploiting Intra-domain and Inter-domain Knowledge with Reliable Transfer for Cardiac Segmentation