Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encoding

Jiajun Zhu, Peihao Wang, Ruisi Cai +3

Transformers rely on both content-based and position-based addressing mechanisms to make predictions, but existing positional encoding techniques often diminish the effectiveness o…