Transformers Learn Two Competing Sorting Algorithms, and Only One Length-Generalizes

Khashayar Gatmiry, Nathan W. Henry, Chloe H. Su +5

We show that two-layer transformers trained to near-zero loss on a sorting task converge to one of two mechanistically distinct solution classes depending on their initialization.…