$\pi$-Attention: Online Efficient Sparse Transformers for Long-Context Modeling

Dong Liu, Pike D. Liu, Chang Liu +1

Sparse attention is crucial in long-context Transformers, which restricts each token to a limited neighborhood and thereby reduces the quadratic cost of full self-attention. Local…