Norm$\times$Direction: Restoring the Missing Query Norm in Vision Linear Attention

Weikang Meng, Yadan Luo, Liangyu Huo +4

Linear attention mitigates the quadratic complexity of softmax attention but suffers from a critical loss of expressiveness. We identify two primary causes: (1) The normalization o…