Bridging the Divide: Reconsidering Softmax and Linear Attention

Dongchen Han, Yifan Pu, Zhuofan Xia +6

Widely adopted in modern Vision Transformer designs, Softmax attention can effectively capture long-range visual information; however, it incurs excessive computational cost when d…