Distilling to Hybrid Attention Models via KL-Guided Layer Selection

Yanhong Li, Songlin Yang, Shawn Tan +4

Distilling pretrained softmax attention Transformers into more efficient hybrid architectures that interleave softmax and linear attention layers is a promising approach for improv…