1 paper
Shenzhi Yang, Guangcheng Zhu, Bowen Song +8
On-policy distillation (OPD) supervises the student exclusively in the output space by matching next-token distributions. This paradigm suffers from two limitations: (i) a high-var…