Memorization Dynamics in Knowledge Distillation for Language Models

Jaydeep Borkar, Karan Chadha, Niloofar Mireshghallah +6

Knowledge Distillation (KD) is increasingly adopted to transfer capabilities from large language models to smaller ones, offering significant improvements in efficiency and utility…