Why Softmax Attention Outperforms Linear Attention

Yichuan Deng, Zhao Song, Kaijun Yuan +1

Large transformer models have achieved state-of-the-art results in numerous natural language processing tasks. Among the pivotal components of the transformer architecture, the att…