Beyond Absolute Imitation: Anchored Residual Guidance for Privileged On-Policy Distillation

Wenhao Zhang

On-policy distillation (OPD) has demonstrated strong empirical gains in enhancing complex reasoning in LLMs by aligning a student model with a teacher's predictive distribution ove…