Kinetic theory for Transformers and the lost-in-the-middle phenomenon

Mitia Duerinckx, Borjan Geshkovski, Stefano Rossi

We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansion…