E${}^2$-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation

Yifei Liu, Minghao Fang, Xinyu Gu +6

On-policy self-distillation (OPSD) provides dense token-level supervision without a second model: one network acts as teacher with the reference solution and as student with only t…