Learnable Low-Rank Polynomial Sketch for Effective Linear Attention

Zongyue Qin, Haoran Deng, Jason Cong +1

Softmax attention in Transformers suffers from quadratic complexity in sequence length, making it impractical for long-context applications. Linear attention alleviates this issue…