Mesh-Attention: A New Communication-Efficient Distributed Attention with Improved Data Locality

Sirui Chen, Jingji Chen, Siqi Zhu +3

Distributed attention is essential for scaling large language models (LLMs) to long contexts, yet existing methods either have limited parallelism or incur high communication costs…