9 citations · 10 across the 20 of their papers we have counts for
20 papers
Interpretability for Turing Machines
Billy Snikkers, Rumi Salazar, Daniel Murfet +1
We show that susceptibilities, an interpretability technique developed for neural networks, can identify the presence of algorithmic structure in Turing machines by probing the loc…
Patterning in Practice: Debiasing Reward Models with Susceptibilities
George Wang, Elizabeth Donoway, Daniel Murfet
Reward models trained on human preferences are known to suffer from length, formatting, and other stylistic biases. In this paper we use patterning, which reweights each preference…
Elicitation without Backpropagation: Steering Model Behavior by Optimizing the Latent Posterior
Garrett Baker, Vinayak Pathak, Daniel Murfet +1
In the \emph{latent posterior model} of transformer behavior, the next-token distribution arises from a posterior over latent predictive models conditioned on the context, mixed to…
Interpreting Reinforcement Learning Agents with Susceptibilities
Chris Elliott, Einar Urdshals, David Quarel +1
Susceptibilities are a technique for neural network interpretability that studies the response of posterior expectation values of observables to perturbations of the loss. We gener…
Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learning
Chris Elliott, Daniel Murfet
These notes introduce the theory of susceptibilities as developed in [arXiv:2504.18274, arXiv:2601.12703] for interpreting neural networks. The susceptibility of an observable …
Linear Response Estimators for Singular Statistical Models
Chris Elliott, Daniel Murfet
We define susceptibilities as a measure of the response of an observable quantity of a parameterized statistical model to a perturbation of the data for a general class of observab…
Stagewise Reinforcement Learning and the Geometry of the Regret Landscape
Chris Elliott, Einar Urdshals, David Quarel +2
Singular learning theory characterizes Bayesian learning as an evolving tradeoff between accuracy and complexity, with transitions between qualitatively different solutions as samp…
Patterning: The Dual of Interpretability
George Wang, Daniel Murfet
Mechanistic interpretability aims to understand how neural networks generalize beyond their training data by reverse-engineering their internal structures. We introduce patterning…
Towards Spectroscopy: Susceptibility Clusters in Language Models
Andrew Gordon, Garrett Baker, George Wang +3
Spectroscopy infers the internal structure of physical systems by measuring their response to perturbations. We apply this principle to neural networks: perturbing the data distrib…
Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory
Einar Urdshals, Edmund Lau, Jesse Hoogland +2
We study neural network compressibility by using singular learning theory to extend the minimum description length (MDL) principle to singular models like neural networks. Through…
Embryology of a Language Model
George Wang, Garrett Baker, Andrew Gordon +1
Understanding how language models develop their internal computational structure is a central problem in the science of deep learning. While susceptibilities, drawn from statistica…
Modes of Sequence Models and Learning Coefficients
Zhongtian Chen, Daniel Murfet
We develop a geometric account of sequence modelling that links patterns in the data to measurable properties of the loss landscape in transformer networks. First, we cast conditio…
Programs as Singularities
Daniel Murfet, Will Troiani
We develop a correspondence between the structure of Turing machines and the structure of singularities of real analytic functions, based on connecting the Ehrhard-Regnier derivati…
Structural Inference: Interpreting Small Language Models with Susceptibilities
Garrett Baker, George Wang, Jesse Hoogland +1
We develop a linear response framework for interpretability that treats a neural network as a Bayesian statistical mechanical system. A small perturbation of the data distribution,…
You Are What You Eat -- AI Alignment Requires Understanding How Data Shapes Structure and Generalisation
Simon Pepin Lehalleur, Jesse Hoogland, Matthew Farrugia-Roberts +5
In this position paper, we argue that understanding the relation between structure in the data distribution and structure in trained models is central to AI alignment. First, we di…
Dynamics of Transient Structure in In-Context Linear Regression Transformers
Liam Carroll, Jesse Hoogland, Matthew Farrugia-Roberts +1
Modern deep neural networks display striking examples of rich internal computational structure. Uncovering principles governing the development of such structure is a priority for…
Open Problems in Mechanistic Interpretability
Lee Sharkey, Bilal Chughtai, Joshua Batson +26
Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks' capabilities in order to accomplish concrete scientific and engineering goa…
Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient
George Wang, Jesse Hoogland, Stan van Wingerden +2
We introduce refined variants of the Local Learning Coefficient (LLC), a measure of model complexity grounded in singular learning theory, to study the development of internal stru…
Loss Landscape Degeneracy and Stagewise Development in Transformers
Jesse Hoogland, George Wang, Matthew Farrugia-Roberts +3
Deep learning involves navigating a high-dimensional loss landscape over the neural network parameter space. Over the course of training, complex computational structures form and…
Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition
Zhongtian Chen, Edmund Lau, Jake Mendel +2
We investigate phase transitions in a Toy Model of Superposition (TMS) using Singular Learning Theory (SLT). We derive a closed formula for the theoretical loss and, in the case of…