Attention Mechanisms Through the Lens of Numerical Methods: Approximation Methods and Alternative Formulations

Michel Fabrice Serret, Alice Cortinovis, Yijun Dong +10

The attention mechanism is the computational core of modern Transformer architectures, but its quadratic complexity in the input sequence length is the bottleneck for large-scale i…