Self-attention as an attractor network: transient memories without backpropagation

Francesco D'Amico, Matteo Negri

Transformers are one of the most successful architectures of modern neural networks. At their core there is the so-called attention mechanism, which recently interested the physics…