Uniform Scaling Limits in AdamW-Trained Transformers

William Gibson, Christoph Reisinger

We study the large-depth limit of transformers trained with AdamW, by modelling the hidden-state dynamics as an interacting particle system (IPS) coupled through the attention mech…