Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention

Qucheng Gao, Zuyi Yang, Xiao Chen

Transformer layers generate state-dependent interaction networks: token representations determine the attention matrix, which in turn updates the representations. We study this fee…