Making Every Head Count: Sparse Attention Without the Speed-Performance Trade-off

Mingkuan Zhao, Wentao Hu, Jiayin Wang +5

The design of Large Language Models (LLMs) has long been hampered by a fundamental conflict within their core attention mechanism: its remarkable expressivity is built upon a compu…