Optimizing Knowledge Distillation in Transformers: Enabling Multi-Head Attention without Alignment Barriers

Zhaodong Bing, Linze Li, Jiajun Liang

Knowledge distillation (KD) in transformers often faces challenges due to misalignment in the number of attention heads between teacher and student models. Existing methods either…