BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences

Ao Sun, Weilin Zhao, Xu Han +4

Effective attention modules have played a crucial role in the success of Transformer-based large language models (LLMs), but the quadratic time and memory complexities of these att…