2 citations · 4 across the 8 of their papers we have counts for
9 papers
The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment
Gonçalo Paulo, Louis Jaburi, Nora Belrose +2
Fine-tuning large language models on narrow, misaligned tasks can undo their post-training alignment and induce novel misaligned behaviors -- a phenomenon known as \emph{emergent m…
Capability Provenance in Language Models: A Case Study in Social Reasoning
Glenn Matlin, Chandreyi Chakraborty, Saehee Eom +8
We use training-data attribution as an interpretable tool for capability discovery, mapping which regions of the pretraining corpus support social reasoning versus STEM reasoning i…
Bergson: An Open Source Library for Data Attribution
Lucia Quirke, Louis Jaburi, David Johnston +6
Data attribution is a promising field in interpretability that aims to explain model behavior through the influence of its training data, with applications including debugging unde…
Binary Sparse Coding for Interpretability
Lucia Quirke, Stepan Shabalin, Nora Belrose
Sparse autoencoders (SAEs) are used to decompose neural network activations into sparsely activating features, but many SAE features are only interpretable at high activation stren…
Slowing Learning by Erasing Simple Features
Lucia Quirke, Nora Belrose
Prior work suggests that neural networks tend to learn low-order moments of the data distribution first, before moving on to higher-order correlations. In this work, we derive a no…
Neural Networks Learn Statistics of Increasing Complexity
Nora Belrose, Quintin Pope, Lucia Quirke +2
The distributional simplicity bias (DSB) posits that neural networks learn low-order moments of the data distribution first, before moving on to higher-order correlations. In this…
Structured World Representations in Maze-Solving Transformers
Michael Igorevich Ivanitskiy, Alex F. Spies, Tilman Räuker +9
Transformer models underpin many recent advances in practical machine learning applications, yet understanding their internal behavior continues to elude researchers. Given the siz…
Training Dynamics of Contextual N-Grams in Language Models
Lucia Quirke, Lovis Heindrich, Wes Gurnee +1
Prior work has shown the existence of contextual neurons in language models, including a neuron that activates on German text. We show that this neuron exists within a broader cont…
A Configurable Library for Generating and Manipulating Maze Datasets
Michael Igorevich Ivanitskiy, Rusheb Shah, Alex F. Spies +8
Understanding how machine learning models respond to distributional shifts is a key research challenge. Mazes serve as an excellent testbed due to varied generation algorithms offe…