Trainable Dynamic Mask Sparse Attention

Jingze Shi, Yifan Wu, Yiran Peng +4

The increasing demand for long-context modeling in large language models (LLMs) is bottlenecked by the quadratic complexity of the standard self-attention mechanism. The community…