1 citations · 1 across the 5 of their papers we have counts for
7 papers
A Step Towards Forgetting: Optimiser History and the Loss of Answer Mass
Valeria Ruscio, Seth Nabarro, Keiran Thompson
During fine-tuning, a language model can assign less probability to previously learned answers even when the current gradient acts to preserve that probability. With momentum, each…
Where does Absolute Position come from in decoder-only Transformers?
Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri
RoPE-trained transformers distinguish absolute position in their attention patterns, even though RoPE encodes only relative offsets in the inner product. We trace this leakage to t…
Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space
Valeria Ruscio, Eli-Shaoul Khedouri, Keiran Thompson
Cross-entropy pretraining and preference alignment update the same transformer weights, but leave geometrically distinct traces. We characterise this asymmetry with a relative-subs…
On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decode
Valeria Ruscio, Keiran Thompson
A language model can give the wrong answer even when the correct answer is decodable from its intermediate states. To study this gap between decodability and selection, we distingu…
What are you sinking? A geometric approach on attention sink
Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri
Attention sink (AS) is a consistent pattern in transformer attention maps where certain tokens (often special tokens or positional anchors) disproportionately attract attention fro…
Beyond Position: the emergence of wavelet-like properties in Transformers
Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri
This paper studies how Transformer models with Rotary Position Embeddings (RoPE) develop emergent, wavelet-like properties that compensate for the positional encoding's theoretical…
Attention-likelihood relationship in transformers
Valeria Ruscio, Valentino Maiorca, Fabrizio Silvestri
We analyze how large language models (LLMs) represent out-of-context words, investigating their reliance on the given context to capture their semantics. Our likelihood-guided text…