Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization

Jiaqi Tian, Ting Long

On-Policy Self-distillation is a self-improvement paradigm that enables large language models (LLMs) to improve their capabilities by leveraging the same model as both teacher and…