Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformers

Jia Liang, Xi Jin, Liangming Pan

Looped Transformers can generalize to reasoning chains longer than those encountered during training, but the computations enabling this behavior and limiting its extent remain unc…