On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspective

Gabriel Mongaras, Eric C. Larson

Since its introduction, softmax attention has become the backbone of modern transformer architectures due to its expressiveness and scalability across a wide range of tasks. Howeve…