5 papers
Where does Absolute Position come from in decoder-only Transformers?
Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri
RoPE-trained transformers distinguish absolute position in their attention patterns, even though RoPE encodes only relative offsets in the inner product. We trace this leakage to t…
Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space
Valeria Ruscio, Eli-Shaoul Khedouri, Keiran Thompson
Cross-entropy pretraining and preference alignment update the same transformer weights, but leave geometrically distinct traces. We characterise this asymmetry with a relative-subs…
The Phenomenology of Hallucinations
Valeria Ruscio, Keiran Thompson
We show that language models hallucinate not because they fail to detect uncertainty, but because of a failure to integrate it into output generation. Across architectures, uncerta…
What are you sinking? A geometric approach on attention sink
Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri
Attention sink (AS) is a consistent pattern in transformer attention maps where certain tokens (often special tokens or positional anchors) disproportionately attract attention fro…
Beyond Position: the emergence of wavelet-like properties in Transformers
Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri
This paper studies how Transformer models with Rotary Position Embeddings (RoPE) develop emergent, wavelet-like properties that compensate for the positional encoding's theoretical…