MHLA: Restoring Expressivity of Linear Attention via Token-Level Multi-Head

Kewei Zhang, Ye Huang, Yufan Deng +5

While the Transformer architecture dominates many fields, its quadratic self-attention complexity hinders its use in large-scale applications. Linear attention offers an efficient…