29 citations · 50 across the 20 of their papers we have counts for
10 papers · 1 filter
Binary Sparse Coding for Interpretability
Lucia Quirke, Stepan Shabalin, Nora Belrose
Sparse autoencoders (SAEs) are used to decompose neural network activations into sparsely activating features, but many SAE features are only interpretable at high activation stren…
Evaluating SAE interpretability without explanations
Gonçalo Paulo, Nora Belrose
Sparse autoencoders (SAEs) and transcoders have become important tools for machine learning interpretability. However, measuring how interpretable they are remains challenging, wit…
Mechanistic Anomaly Detection for "Quirky" Language Models
David O. Johnston, Arkajyoti Chakraborty, Nora Belrose
As LLMs grow in capability, the task of supervising LLMs becomes more challenging. Supervision failures can occur if LLMs are sensitive to factors that supervisors are unaware of.…
Examining Two Hop Reasoning Through Information Content Scaling
David Johnston, Nora Belrose
Prior work has found that transformers have an inconsistent ability to learn to answer latent two-hop questions -- questions of the form "Who is Bob's mother's boss?" We study why…
Transcoders Beat Sparse Autoencoders for Interpretability
Gonçalo Paulo, Stepan Shabalin, Nora Belrose
Sparse autoencoders (SAEs) extract human-interpretable features from deep neural networks by transforming their activations into a sparse, higher dimensional latent space, and then…
Slowing Learning by Erasing Simple Features
Lucia Quirke, Nora Belrose
Prior work suggests that neural networks tend to learn low-order moments of the data distribution first, before moving on to higher-order correlations. In this work, we derive a no…
Converting MLPs into Polynomials in Closed Form
Nora Belrose, Alice Rigg
Recent work has shown that purely quadratic functions can replace MLPs in transformers with no significant loss in performance, while enabling new methods of interpretability based…
Partially Rewriting a Transformer in Natural Language
Gonçalo Paulo, Nora Belrose
The greatest ambition of mechanistic interpretability is to completely rewrite deep neural networks in a format that is more amenable to human understanding, while preserving their…
Sparse Autoencoders Trained on the Same Data Learn Different Features
Gonçalo Paulo, Nora Belrose
Sparse autoencoders (SAEs) are a useful tool for uncovering human-interpretable features in the activations of large language models (LLMs). While some expect SAEs to find the true…
Estimating the Probability of Sampling a Trained Neural Network at Random
Adam Scherlis, Nora Belrose
We present and analyze an algorithm for estimating the size, under a Gaussian or uniform measure, of a localized neighborhood in neural network parameter space with behavior simila…