A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions

Luyang Fang, Haoran Lu, Jiazhang Cai +4

Knowledge Distillation (KD) has emerged as a vital paradigm for transferring the capabilities of high-capacity models to efficient ``student'' counterparts, addressing critical cha…