Algebraic Decomposition Theory for Transformer Length Generalization

Andy Yang, Blerta Veseli, Corentin Barloy +5

Transformer-based language models are known to sometimes generalize to sequences longer than seen during training, but we lack a precise characterization of which tasks admit lengt…