Long-Context Attention Benchmark: From Kernel Efficiency to Distributed Context Parallelism

Tao Bu, Qiangang Wang, Bowen Zeng +4

Transformer-based large language models (LLMs) have achieved remarkable success, yet their standard attention mechanism incurs quadratic computation and memory costs with respect t…