A Formal Framework for Understanding Length Generalization in Transformers

Xinting Huang, Andy Yang, Satwik Bhattamishra +5

A major challenge for transformers is generalizing to sequences longer than those observed during training. While previous works have empirically shown that transformers can either…