Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models

Junjie Yang, Junhao Song, Xudong Han +9

Knowledge distillation (KD) is a technique for transferring knowledge from complex teacher models to simpler student models, significantly enhancing model efficiency and accuracy.…