Distance generalization in transformers: why bother with positional encoding?

Daniel Henrik Nevermann, Claudius Gros

Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. Here we focus on distance gener…