Enhancing Logits Distillation with Plug\&Play Kendall's $\tau$ Ranking Loss

Yuchen Guan, Runxi Cheng, Kang Liu +1

Knowledge distillation typically minimizes the Kullback-Leibler (KL) divergence between teacher and student logits. However, optimizing the KL divergence can be challenging for the…