Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Points

Aditya Varre, Gizem Yüce, Nicolas Flammarion

Motivated by empirical observations of prolonged plateaus and stage-wise progression during training, we investigate the loss landscape of transformer models trained on in-context…