Infinite Limits of Multi-head Transformer Dynamics

Blake Bordelon, Hamza Tahir Chaudhry, Cengiz Pehlevan

In this work, we analyze various scaling limits of the training dynamics of transformer models in the feature learning regime. We identify the set of parameterizations that admit w…