One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context

Yingzi Ma, Zichen Zhu, Ming Jiang +1

On-policy self-distillation (OPSD) teaches large language models new skills through a teacher that shares the student's backbone and supervises its own rollouts. Existing teachers…