The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training

Matteo Saponati, Pascal Sager, Pau Vilimelis Aceituno +2

Self-attention is essential to Transformer architectures, yet how information is embedded in the self-attention matrices and how different objective functions impact this process r…