Understanding the Staged Dynamics of Transformers in Learning Latent Structure

Rohan Saha, Farzane Aminmansour, Alona Fyshe

Language modeling has shown us that transformers can discover latent structure from context, but the dynamics of how they acquire different components of that structure remain poor…