On the Generalization of Knowledge Distillation: An Information-Theoretic View

Bingying Li, Haiyun He

Knowledge distillation is widely used to improve generalization in practice, yet its theoretical understanding remains elusive. In the standard distillation setting, a teacher mode…