The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

Piotr Nawrot, Robert Li, Renjie Huang +3

Sparse attention offers a promising strategy to extend long-context capabilities in Transformer LLMs, yet its efficiency-accuracy trade-offs remain unclear due to the lack of compr…