Scaling Linear Attention with Sparse State Expansion

Yuqi Pan, Yongqi An, Zheng Li +6

The Transformer architecture, despite its widespread success, struggles with long-context scenarios due to quadratic computation and linear memory growth. While various linear atte…