How Is Uncertainty Propagated in Knowledge Distillation?

Ziyao Cui, Jian Pei

Knowledge distillation transfers behavior from a teacher to a student model, but the process is inherently stochastic: teacher outputs, student training, and student inference can…