SAGA: Selective Adaptive Gating for Efficient and Expressive Linear Attention

Yuan Cao, Dong Wang

While Transformer architecture excel at modeling long-range dependencies contributing to its widespread adoption in vision tasks the quadratic complexity of softmax-based attention…