Solving Without Stopping: On-Policy Distillation at Small Scale

Hongyang Li, Yiming Zhu, Xiao Li +3

On-policy distillation, where a student learns from a stronger teacher's feedback on its own outputs, is a common way to pass reasoning to smaller models. We analyze what it transf…