Continually Post-training Small Models via Synthetic Self-distillation

Woosung Koh, Hyunsoo Lee, Kyungjae Lee +4

Continual post-training faces a dual challenge of acquisition and retention: models must learn new capabilities while preserving prior knowledge. While Self-Distillation Fine-Tunin…