Group Relative Knowledge Distillation: Learning from Teacher's Relational Inductive Bias

Chao Li, Changhua Zhou, Jia Chen

Knowledge distillation typically transfers knowledge from a teacher model to a student model by minimizing differences between their output distributions. However, existing distill…