3 papers
cs.AI2026
Where does Absolute Position come from in decoder-only Transformers?
Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri
RoPE-trained transformers distinguish absolute position in their attention patterns, even though RoPE encodes only relative offsets in the inner product. We trace this leakage to t…
cs.LG2025
What are you sinking? A geometric approach on attention sink
Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri
Attention sink (AS) is a consistent pattern in transformer attention maps where certain tokens (often special tokens or positional anchors) disproportionately attract attention fro…
cs.LG2025
Beyond Position: the emergence of wavelet-like properties in Transformers
Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri
This paper studies how Transformer models with Rotary Position Embeddings (RoPE) develop emergent, wavelet-like properties that compensate for the positional encoding's theoretical…