391 citations · 838 across the 62 of their papers we have counts for
16 papers · 1 filter
BatchTopK Sparse Autoencoders
Bart Bussmann, Patrick Leask, Neel Nanda
Sparse autoencoders (SAEs) have emerged as a powerful tool for interpreting language model activations by decomposing them into sparse, interpretable features. A popular approach i…
Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks
Adam Karvonen, Can Rager, Samuel Marks +1
Sparse Autoencoders (SAEs) are an interpretability technique aimed at decomposing neural network activations into interpretable units. However, a major bottleneck for SAE developme…
Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models
Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan +1
Hallucinations in large language models are a widespread problem, yet the mechanisms behind whether models will hallucinate are poorly understood, limiting our ability to solve thi…
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy +7
Sparse autoencoders (SAEs) are an unsupervised method for learning a sparse decomposition of a neural network's latent representations into seemingly interpretable features. Despit…
Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat +4
Sparse autoencoders (SAEs) are a promising unsupervised approach for identifying causally relevant and interpretable linear features in a language model's (LM) activations. To be u…
Interpreting Attention Layer Outputs with Sparse Autoencoders
Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom +2
Decomposing model activations into interpretable components is a key open problem in mechanistic interpretability. Sparse autoencoders (SAEs) are a popular method for decomposing t…
Confidence Regulation Neurons in Language Models
Alessandro Stolfo, Ben Wu, Wes Gurnee +4
Despite their widespread use, the mechanisms by which large language models (LLMs) represent and regulate uncertainty in next-token predictions remain largely unexplored. This stud…
Transcoders Find Interpretable LLM Feature Circuits
Jacob Dunefsky, Philippe Chlenski, Neel Nanda
A key goal in mechanistic interpretability is circuit analysis: finding sparse subgraphs of models corresponding to specific behaviors or capabilities. However, MLP sublayers make…
Refusal in Language Models Is Mediated by a Single Direction
Andy Arditi, Oscar Obeso, Aaquib Syed +4
Conversational large language models are fine-tuned for both instruction-following and safety, resulting in models that obey benign requests but refuse harmful ones. While this ref…
Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control
Aleksandar Makelov, George Lange, Neel Nanda
Disentangling model activations into meaningful features is a central problem in interpretability. However, the absence of ground-truth for these features in realistic scenarios ma…
Improving Dictionary Learning with Gated Sparse Autoencoders
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith +5
Recent work has found that sparse autoencoders (SAEs) are an effective technique for unsupervised discovery of interpretable features in language models' (LMs) activations, by find…
How to use and interpret activation patching
Stefan Heimersheim, Neel Nanda
Activation patching is a popular mechanistic interpretability technique, but has many subtleties regarding how it is applied and how one may interpret the results. We provide a sum…
AtP*: An efficient and scalable method for localizing LLM behaviour to components
János Kramár, Tom Lieberum, Rohin Shah +1
Activation Patching is a method of directly computing causal attributions of behavior to model components. However, applying it exhaustively requires a sweep with cost scaling line…
Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs
Bilal Chughtai, Alan Cooney, Neel Nanda
How do transformer-based large language models (LLMs) store and retrieve knowledge? We focus on the most basic form of this task -- factual recall, where the model is tasked with e…
Explorations of Self-Repair in Language Models
Cody Rushing, Neel Nanda
Prior interpretability research studying narrow distributions has preliminarily identified self-repair, a phenomena where if components in large language models are ablated, later…
Universal Neurons in GPT2 Language Models
Wes Gurnee, Theo Horsley, Zifan Carl Guo +5
A basic question within the emerging field of mechanistic interpretability is the degree to which neural networks learn the same underlying mechanisms. In other words, are neural m…