Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation

Yongkang Yang, Zhezheng Hao, Hong Zhang +8

On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters through self-distillation. Two recent research…