On the Limitations and Capabilities of Position Embeddings for Length Generalization

Yang Chen, Yitao Liang, Zhouchen Lin

In Transformers, Position Embeddings (PEs) significantly influence Length Generalization (LG) performance, yet their fundamental role remains unclear. In this work, we investigate…