Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

Jacqueline He, Howard Yen, Shuyue Stella Li +9

Logit-based knowledge distillation (KD) is used to train smaller language models (LMs) via supervision from stronger teachers, but whether its benefits are consistent across traini…