2 papers
cs.CL2026
Beyond Imitation: Reflective On-Policy Self-Distillation for LLM Reasoning
Ziqi Zhao, Xinyu Ma, Liu Yang +6
On-policy self-distillation (OPSD) improves the reasoning capabilities of large language models (LLMs) by providing dense token-level supervision for on-policy rollouts. However, e…
cs.CL2025
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
Jitai Hao, Qiang Huang, Hao Liu +3
Training high-performing Small Language Models (SLMs) remains costly, even with knowledge distillation and pruning from larger teacher models. Existing work often faces three key c…