Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers

Noboru Isobe, Daisuke Inoue, Masaaki Imaizumi

Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theor…