On Transformer Dynamics

Abstract

We develop a geometric framework in which the token dynamics of a transformer are modeled by a system of interacting particles on a Riemannian manifold $\mathcal{M}$, the attention mechanism being encoded by a time-independent two-body interaction law, that is, a section of the pullback bundle $\ddot{I}{€}_2^{\ast }(T\mathcal{M})$ over $\mathcal{M}\times \mathcal{M}$. Within this framework we isolate two features that a family of interaction laws must possess in order to model language: it must realize generic nonlocal and nonreciprocal forces, and it must parametrize vector fields on a high-dimensional manifold efficiently. We show that both features are achieved simultaneously in a transformer model. Our main theorem produces a finitely parametrized family of interaction laws that is \emph{universal}: it realizes an arbitrary prescribed attention digraph. Moreover, we show that the cost of realizing a given attention digraph is governed by two combinatorial invariants of the digraph, namely its biclique cover number, which we identify with the least number of hubs in a hub extension, and its hub-chromatic index.