Consistently Informative Soft-Label Temperature for Knowledge Distillation

Hoang-Chau Luong, Nghia Van Vo, Kaiqi Zhao +1

Knowledge distillation (KD) transfers knowledge from a high-capacity teacher to a compact student by matching their predictive distributions, with temperature scaling serving as a…