Self-Attention And Beyond the Infinite: Towards Linear Transformers with Infinite Self-Attention

Giorgio Roffo, Hazem Abdelkawy, Nilli Lavie +1

The quadratic cost of softmax attention limits Transformer scalability in high-resolution vision. We introduce Infinite Self-Attention (InfSA), a spectral reformulation that treats…