7 citations · 11 across the 9 of their papers we have counts for
10 papers
The Surprising Effectiveness of Shared Memory in Looped Transformers
Giovanni Monea, Keshav Ramji, Yousef El-Kurdi +4
Looped Transformers apply the same layers several times per token, adding compute to improve quality without more parameters. Each recursion, however, writes its own key-value cach…
Almost Free State Prediction Separation
John Langford, Nathan Godey, Giovanni Monea +5
State--prediction separation (SPS) relieves a language model's hidden state of two competing burdens---summarizing the context and predicting the next token---by splitting the forw…
The State-Prediction Separation Hypothesis
Giovanni Monea, Nathan Godey, Kianté Brantley +1
Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the \emph{state-prediction sep…
Breadcrumbs Reasoning: Memory-Efficient Reasoning with Compression Beacons
Giovanni Monea, Yair Feldman, Shankar Padmanabhan +2
The scalability of large language models for long-context reasoning is severely constrained by the linear growth of their Transformer key-value cache, which incurs significant memo…
Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers
Clément Dumas, Chris Wendler, Veniamin Veselovsky +2
A central question in multilingual language modeling is whether large language models (LLMs) develop a universal concept representation, disentangled from specific languages. In th…
Controllable Context Sensitivity and the Knob Behind It
Julian Minder, Kevin Du, Niklas Stoehr +4
When making predictions, a language model must trade off how much it relies on its context vs. its prior knowledge. Choosing how sensitive the model is to its context is a fundamen…
LLMs Are In-Context Bandit Reinforcement Learners
Giovanni Monea, Antoine Bosselut, Kianté Brantley +1
Large Language Models (LLMs) excel at in-context learning (ICL), a supervised learning technique that relies on adding annotated examples to the model context. We investigate a con…
Do Llamas Work in English? On the Latent Language of Multilingual Transformers
Chris Wendler, Veniamin Veselovsky, Giovanni Monea +1
We ask whether multilingual language models trained on unbalanced, English-dominated corpora use English as an internal pivot language -- a question of key importance for understan…
A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia
Giovanni Monea, Maxime Peyrard, Martin Josifoski +6
Large language models (LLMs) have an impressive ability to draw on novel information supplied in their context. Yet the mechanisms underlying this contextual grounding remain unkno…
PaSS: Parallel Speculative Sampling
Giovanni Monea, Armand Joulin, Edouard Grave
Scaling the size of language models to tens of billions of parameters has led to impressive performance on a wide range of tasks. At generation, these models are used auto-regressi…