iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs

Ahmed Khaled Khamis, Xiaotong Ji, Hassan Jaber +4

On-policy self-distillation fine-tuning (SDFT) learns new skills from demonstrations while reducing forgetting, but it always distils toward the full demonstration-conditioned teac…