The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models

Ulme Wennberg, Gustav Eje Henter

Mechanisms for encoding positional information are central for transformer-based language models. In this paper, we analyze the position embeddings of existing language models, fin…