Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?

Wenze Lin, Jiyuan Long, Jiale Zhao +11

Since the advent of knowledge distillation, KL divergence has been the standard loss in distillation. Recently, on-policy distillation (OPD) has emerged as an efficient post-traini…