1 citations · 1 across the 5 of their papers we have counts for
3 papers · 1 filter
The Geometry of Learning: Read-Write Asymmetry Across Training
Valeria Ruscio, Eli-Shaoul Khedouri, Keiran Thompson
Pretraining and alignment update the same transformer weights, and we ask in which directions each stage changes them. The residual stream gives every matrix a shared coordinate sy…
What are you sinking? A geometric approach on attention sink
Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri
Attention sink (AS) is a consistent pattern in transformer attention maps where certain tokens (often special tokens or positional anchors) disproportionately attract attention fro…
Beyond Position: the emergence of wavelet-like properties in Transformers
Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri
This paper studies how Transformer models with Rotary Position Embeddings (RoPE) develop emergent, wavelet-like properties that compensate for the positional encoding's theoretical…