Spectral Selection in Symmetric Self-Attention Dynamics

Christian Kuehn, Jaeyoung Yoon

We study self-attention dynamics on the unit sphere as an interacting particle system arising from an idealized Transformer-type update. Under a symmetry assumption on weight matri…