Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding

Zhigeng Liu, Zhiyuan Ning, Ruixiao Li +5

The development of long-context Large Language Models (LLMs) is constrained by the memory bandwidth bottleneck and quadratic complexity of the attention mechanism during decoding.…