184 citations · 489 across the 31 of their papers we have counts for
32 papers
Compressed Computation under Loss is likely Computation in Superposition
Francisco Ferreira da Silva, Stefan Heimersheim
Neural networks are thought to represent concepts as directions in their activation space, and superposition lets them encode more concepts than they have dimensions. It is natural…
Individual Parameters in Weight-Sparse Transformers Appear Interpretable
Arnau Marin-Llobet, Stefan Heimersheim
A central goal of mechanistic interpretability is to understand how neural networks work and what each individual component does. Dominant circuit-finding approaches focus on a spe…
The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology
Andrzej Szablewski, Gabriel Konar-Steenberg, Raffaello Fornasiere +2
Model organisms (MOs) - language models trained to exhibit undesired or unnatural behaviours - are frequently used as testbeds for evaluating white-box interpretability techniques.…
Evidence for feature-specific error correction in LLMs
Francisco Ferreira da Silva, Stefan Heimersheim
Understanding the features of large language models (LLMs) is a central goal of interpretability. LLMs are commonly assumed to use superposition to represent more features than the…
Compressed Computation is (probably) not Computation in Superposition
Jai Bhagat, Sara Molas-Medina, Giorgi Giglemiani +1
We study whether the Compressed Computation (CC) toy model (Braun et al., 2025) is an instance of computation in superposition. The CC model appears to compute 100 ReLU functions w…
Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution
Matthew Kowal, Goncalo Paulo, Louis Jaburi +6
As large language models are increasingly trained and fine-tuned, practitioners need methods to identify which training data drive specific behaviors, particularly unintended ones.…
The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probes
Mohammad Taufeeque, Stefan Heimersheim, Adam Gleave +1
Training against white-box deception detectors has been proposed as a way to make AI systems honest. However, such training risks models learning to obfuscate their deception to ev…
SCALAR: Benchmarking SAE Interaction Sparsity in Toy LLMs
Sean P. Fillingham, Andrew Gordon, Peter Lai +3
Mechanistic interpretability aims to decompose neural networks into interpretable features and map their connecting circuits. The standard approach trains sparse autoencoders (SAEs…
Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability
Luca Baroni, Galvin Khara, Joachim Schaeffer +2
Layer-wise normalization (LN) is an essential component of virtually all transformer-based large language models. While its effects on training stability are well documented, its r…
Benchmarking Deception Probes via Black-to-White Performance Boosts
Avi Parrack, Carlo Leonardo Attubato, Stefan Heimersheim
AI assistants will occasionally respond deceptively to user queries. Recently, linear classifiers (called "deception probes") have been trained to distinguish the internal activati…
Rapid and Late Cosmic Reionization Driven by Massive Galaxies: a Joint Analysis of Constraints from 21-cm, Lyman Line & CMB Data Sets
Peter H. Sims, Harry T. J. Bevins, Anastasia Fialkov +6
Observations of the Epoch of Reionization (EoR) have the potential to answer long-standing questions of astrophysical interest regarding the nature of the first luminous sources an…
Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition
Dan Braun, Lucius Bushnaq, Stefan Heimersheim +2
Mechanistic interpretability aims to understand the internal mechanisms learned by neural networks. Despite recent progress toward this goal, it remains unclear how best to decompo…
Detecting Strategic Deception Using Linear Probes
Nicholas Goldowsky-Dill, Bilal Chughtai, Stefan Heimersheim +1
AI models might use deceptive strategies as part of scheming or misaligned behaviour. Monitoring outputs alone is insufficient, since the AI might produce seemingly benign outputs…
Open Problems in Mechanistic Interpretability
Lee Sharkey, Bilal Chughtai, Joshua Batson +26
Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks' capabilities in order to accomplish concrete scientific and engineering goa…
Investigating Sensitive Directions in GPT-2: An Improved Baseline and Comparative Analysis of SAEs
Daniel J. Lee, Stefan Heimersheim
Sensitive directions experiments attempt to understand the computational features of Language Models (LMs) by measuring how much the next token prediction probabilities change by p…
Evolution of SAE Features Across Layers in LLMs
Daniel Balcells, Benjamin Lerner, Michael Oesterle +2
Sparse Autoencoders for transformer-based language models are typically defined independently per layer. In this work we analyze statistical relationships between features in adjac…
Characterizing stable regions in the residual stream of LLMs
Jett Janiak, Jacek Karwowski, Chatrik Singh Mangat +3
We identify stable regions in the residual stream of Transformers, where the model's output remains insensitive to small activation changes, but exhibits high sensitivity at region…
Evaluating Synthetic Activations composed of SAE Latents in GPT-2
Giorgi Giglemiani, Nora Petrova, Chatrik Singh Mangat +2
Sparse Auto-Encoders (SAEs) are commonly employed in mechanistic interpretability to decompose the residual stream into monosemantic SAE latents. Recent work demonstrates that pert…
You can remove GPT2's LayerNorm by fine-tuning
Stefan Heimersheim
The LayerNorm (LN) layer in GPT-style transformer models has long been a hindrance to mechanistic interpretability. LN is a crucial component required to stabilize the training of…
The Local Interaction Basis: Identifying Computationally-Relevant and Sparsely Interacting Features in Neural Networks
Lucius Bushnaq, Stefan Heimersheim, Nicholas Goldowsky-Dill +7
Mechanistic interpretability aims to understand the behavior of neural networks by reverse-engineering their internal computations. However, current methods struggle to find clear…
Using Degeneracy in the Loss Landscape for Mechanistic Interpretability
Lucius Bushnaq, Jake Mendel, Stefan Heimersheim +5
Mechanistic Interpretability aims to reverse engineer the algorithms implemented by neural networks by studying their weights and activations. An obstacle to reverse engineering ne…
How to use and interpret activation patching
Stefan Heimersheim, Neel Nanda
Activation patching is a popular mechanistic interpretability technique, but has many subtleties regarding how it is applied and how one may interpret the results. We provide a sum…
Constraining the properties of Population III galaxies with multi-wavelength observations
S. Pochinda, T. Gessey-Jones, H. T. J. Bevins +7
The early Universe, spanning 400,000 to 400 million years after the Big Bang (), has been left largely unexplored as the light from luminous objects is too faint t…
FlexKnot and Gaussian Process for 21 cm global signal analysis and foreground separation
Stefan Heimersheim, Leiv Rønneberg, Henry Linton +2
The cosmological 21 cm signal is one of the most promising avenues to study the Epoch of Reionization. One class of experiments aiming to detect this signal is global signal experi…