26 citations · 44 across the 23 of their papers we have counts for
28 papers
A Phenomenology of Neural Geometry in Vision-Language-Action Models
Mozes Jacobs, Aiden Swann, Mathilde Papillon +16
Robotic manipulation is inherently geometric, but what shape does the world take inside the policy that performs it? Prior work on vision-language-action (VLA) policies establishes…
Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models
Aiden Swann, Mozes Jacobs, Mathilde Papillon +12
Can better feature discovery improve control over robot behavior through internal interventions and training data? Recent work in vision and language models reveals multidimensiona…
Causal Analysis of Robot Foundation Models Enables Efficiency Gains
Aiden Swann, Mathilde Papillon, Mozes Jacobs +13
Vision-language-action (VLA) models are a promising approach to general purpose robot manipulation. While the internal computation of vision-language models, which compose the back…
Language Model Activations Inhabit Privileged Error-Correcting Basins
Matthew Finlayson, Francisco Pernice, Eric Todd +16
Language models exhibit remarkable robustness, continuing to produce coherent text even when their activations are perturbed by interventions like linear steering. We hypothesize t…
Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation
Eric Bigelow, Amir Zur, Satchel Grant +7
LLM reasoning is stochastic, and so understanding a model requires grappling with the distribution of reasoning chains that it might produce for a given question, i.e., its uncerta…
Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
Thomas Fel, Matthew Kowal, Mozes Jacobs +22
What is the geometry of a visual percept? The most widely used protocols for decomposing neural network representations into interpretable parts treat concepts as isolated directio…
Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal
Leon Bergen, Usha Bhalla, Sidharth Baskaran +14
Language-model post-training is the main stage at which model behavior is shaped, yet it still largely involves optimization of scalar rewards that summarize diverse desiderata. Th…
Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space
Eric Bigelow, Raphaël Sarfati, Daniel Wurgaft +5
Large Language Models (LLMs) update their behavior in context, which can be viewed as a form of Bayesian inference. However, the structure of the latent hypothesis space over which…
Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
Daniel Wurgaft, Can Rager, Matthew Kowal +13
Neural representations carry rich geometric structure; but does that structure causally shape behavior? To address this question, we intervene along paths through activation space…
Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
Sheridan Feucht, Tal Haklay, Usha Bhalla +9
Does structure in representations imply structure in computation? We study how Llama-3.1-8B reasons over cyclic concepts (e.g., "what month is six months after August?"). Even thou…
Do Sparse Autoencoders Capture Concept Manifolds?
Usha Bhalla, Thomas Fel, Can Rager +9
Sparse autoencoders (SAEs) are widely used to extract interpretable features from neural network representations, often under the implicit assumption that concepts correspond to in…
Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
Siddharth Boppana, Annabel Ma, Max Loeffler +5
We provide evidence of performative chain-of-thought (CoT) in reasoning models, where a model becomes strongly confident in its final answer, but continues generating tokens withou…
Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability
Aaditya Vikram Prasad, Connor Watts, Jack Merullo +4
Language models trained on large-scale datasets have been shown to learn features that encode abstract concepts such as factuality or intent. Such features are traditionally used f…
The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds of Language Models' Posteriors
Raphaël Sarfati, Eric Bigelow, Daniel Wurgaft +6
Large language models (LLMs) form implicit beliefs (posteriors over latent variables) from prompts, but we lack a mechanistic account of how these beliefs are encoded in representa…
From Memorization to Reasoning in the Spectrum of Loss Curvature
Jack Merullo, Srihita Vatsavaya, Lucius Bushnaq +1
We characterize how memorization is represented in transformer models and show that it can be disentangled in the weights of both language models (LMs) and vision transformers (ViT…
Transferring Linear Features Across Language Models With Model Stitching
Alan Chen, Jack Merullo, Alessandro Stolfo +1
In this work, we demonstrate that affine mappings between residual streams of language models is a cheap way to effectively transfer represented features between models. We apply t…
On Linear Representations and Pretraining Data Frequency in Language Models
Jack Merullo, Noah A. Smith, Sarah Wiegreffe +1
Pretraining data has a direct impact on the behaviors and quality of language models (LMs), but we only understand the most basic principles of this relationship. While most work f…
$100K or 100 Days: Trade-offs when Pre-Training with Academic Resources
Apoorv Khandelwal, Tian Yun, Nihal V. Nayak +4
Pre-training is notoriously compute-intensive and academic researchers are notoriously under-resourced. It is, therefore, commonly assumed that academics can't pre-train models. In…
Talking Heads: Understanding Inter-layer Communication in Transformer Language Models
Jack Merullo, Carsten Eickhoff, Ellie Pavlick
Although it is known that transformer language models (LMs) pass features from early layers to later layers, it is not well understood how this information is represented and route…
Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting
Suraj Anand, Michael A. Lepori, Jack Merullo +1
Language models have the ability to perform in-context learning (ICL), allowing them to flexibly adapt their behavior based on context. This contrasts with in-weights learning (IWL…
Axiomatic Causal Interventions for Reverse Engineering Relevance Computation in Neural Retrieval Models
Catherine Chen, Jack Merullo, Carsten Eickhoff
Neural models have demonstrated remarkable performance across diverse ranking tasks. However, the processes and internal mechanisms along which they determine relevance are still l…
Transformer Mechanisms Mimic Frontostriatal Gating Operations When Trained on Human Working Memory Tasks
Aaron Traylor, Jack Merullo, Michael J. Frank +1
Models based on the Transformer neural network architecture have seen success on a wide variety of tasks that appear to require complex "cognitive branching" -- or the ability to m…
Characterizing Mechanisms for Factual Recall in Language Models
Qinan Yu, Jack Merullo, Ellie Pavlick
Language Models (LMs) often must integrate facts they memorized in pretraining with new information that appears in a given context. These two sources can disagree, causing competi…
Circuit Component Reuse Across Tasks in Transformer Language Models
Jack Merullo, Carsten Eickhoff, Ellie Pavlick
Recent work in mechanistic interpretability has shown that behaviors in language models can be successfully reverse-engineered through circuit analysis. A common criticism, however…