Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers

Atsushi Shimizu, Shohei Taniguchi, Yutaka Matsuo

Length generalization is the ability of language models to maintain performance on inputs longer than those seen during pretraining. In this work, we introduce a simple yet powerfu…