Transformers Can Achieve Length Generalization But Not Robustly

Yongchao Zhou, Uri Alon, Xinyun Chen +3

Length generalization, defined as the ability to extrapolate from shorter training sequences to longer test ones, is a significant challenge for language models. This issue persist…