KDFP: A first-principles approach to knowledge distillation in large language models

Ryan Swift, Konstantinos Psounis

Knowledge distillation is an established technique for improving the capabilities of small, efficient student models by training them with the representations of larger, more capab…