1 citations · 1 across the 11 of their papers we have counts for
12 papers
Subliminal Learning Is Steering Vector Distillation
Camila Blank, Agam Bhatia, Senthooran Rajamanoharan +2
Subliminal learning refers to a student language model acquiring a teacher's traits (e.g. a system-prompted preference for owls) when fine-tuned on the teacher's outputs, despite t…
Building Better Activation Oracles
Jan Bauer, Celeste De Schamphelaere, Adam Karvonen +2
Activation Oracles (AOs) are promising methods for interpreting residual stream activations. However, current AOs face important issues, such as hallucinations and vagueness. Addit…
What's the plan? Metrics for implicit planning in LLMs and their application to rhyme generation and question answering
Jim Maar, Denis Paperno, Callum Stuart McDougall +1
Prior work suggests that language models, while trained on next token prediction, show implicit planning behavior: they may select the next token in preparation to a predicted futu…
Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval
Constantin Venhoff, Ashkan Khakzar, Sonia Joseph +2
Training vision language models (VLMs) aims to align visual representations from a vision encoder with the textual representations of a pretrained large language model (LLM). Howev…
Internal states before wait modulate reasoning patterns
Dmitrii Troitskii, Koyena Pal, Chris Wendler +2
Prior work has shown that a significant driver of performance in reasoning models is their ability to reason and self-correct. A distinctive marker in these reasoning traces is the…
Thought Branches: Interpreting LLM Reasoning Requires Resampling
Uzay Macar, Paul C. Bogdan, Senthooran Rajamanoharan +1
Most work interpreting reasoning models studies only a single chain-of-thought (CoT), yet these models define distributions over many possible CoTs. We argue that studying a single…
Base Models Know How to Reason, Thinking Models Learn When
Constantin Venhoff, Iván Arcuschin, Philip Torr +2
What do thinking language models learn during training that their base models lack? We first present an unsupervised method that discovers a model's reasoning behaviors by training…
RelP: Faithful and Efficient Circuit Discovery in Language Models via Relevance Patching
Farnoush Rezaei Jafari, Oliver Eberle, Ashkan Khakzar +1
Activation patching is a standard method in mechanistic interpretability for localizing the components of a model responsible for specific behaviors, but it is computationally expe…
Reasoning-Finetuning Repurposes Latent Representations in Base Models
Jake Ward, Chuqiao Lin, Constantin Venhoff +1
Backtracking, an emergent behavior elicited by reasoning fine-tuning, has been shown to be a key mechanism in reasoning models' enhanced capabilities. Prior work has succeeded in m…
Simple Mechanistic Explanations for Out-Of-Context Reasoning
Atticus Wang, Joshua Engels, Oliver Clive-Griffin +2
Out-of-context reasoning (OOCR) is a phenomenon in which fine-tuned LLMs exhibit surprisingly deep out-of-distribution generalization. Rather than learning shallow heuristics, they…
How Visual Representations Map to Language Feature Space in Multimodal LLMs
Constantin Venhoff, Ashkan Khakzar, Sonia Joseph +2
Effective multimodal reasoning depends on the alignment of visual and linguistic representations, yet the mechanisms by which vision-language models (VLMs) achieve this alignment r…
Because we have LLMs, we Can and Should Pursue Agentic Interpretability
Been Kim, John Hewitt, Neel Nanda +2
The era of Large Language Models (LLMs) presents a new opportunity for interpretability--agentic interpretability: a multi-turn conversation with an LLM wherein the LLM proactively…