The Gradient That Top-k Distillation Never Computes

Rasul Khanbayov, Hasan Kurban

Many deployed distillation pipelines show the student a truncated view of the teacher: keep the top-k probabilities, renormalize, and minimize a divergence on what remains. Renorma…