UNDO: Understanding Distillation as Optimization

Kushal Jain, Piyushi Goyal, Kumar Shridhar

Knowledge distillation has emerged as an effective strategy for compressing large language models' (LLMs) knowledge into smaller, more efficient student models. However, standard o…