Where does Absolute Position come from in decoder-only Transformers?

Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri

RoPE-trained transformers distinguish absolute position in their attention patterns, even though RoPE encodes only relative offsets in the inner product. We trace this leakage to t…