Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation

Qianfeng Yuan, Wenbing Tao

Knowledge distillation transfers knowledge by encouraging a student to match a teacher's predicted class probabilities. These probabilities express not only confidence in the corre…