Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization

Tan Nguyen, Richard G. Baraniuk, Robert M. Kirby +2

Transformers have achieved remarkable success in sequence modeling and beyond but suffer from quadratic computational and memory complexities with respect to the length of the inpu…