More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding

Noam Elata, Itay Lamprecht, Mikey Shechter +3

Autoregressive generation in Large Language Models (LLMs) is constrained by the memory and computational demands of attention mechanisms. Sparse attention methods mitigate this cos…