On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers

Sixu Li, Thomas Jacob Maranzatto, Jan Peszek +5

We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems. In this perspective, tokens are modeled as particles…