Why does Knowledge Distillation Work? Rethink its Attention and Fidelity Mechanism

Chenqi Guo, Shiwei Zhong, Xiaofeng Liu +2

Does Knowledge Distillation (KD) really work? Conventional wisdom viewed it as a knowledge transfer procedure where a perfect mimicry of the student to its teacher is desired. Howe…