9 citations · 24 across the 6 of their papers we have counts for
14 papers
Biases in the Blind Spot: Detecting What LLMs Fail to Mention
Iván Arcuschin, David Chanin, Adrià Garriga-Alonso +1
Large Language Models (LLMs) often provide chain-of-thought (CoT) reasoning traces that appear plausible, but may hide internal biases. We call these unverbalized biases. Monitorin…
Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders
David Chanin, Adrià Garriga-Alonso
Sparse Autoencoders (SAEs) extract features from LLM internal activations, meant to correspond to interpretable concepts. A core SAE training hyperparameter is L0: how many SAE fea…
Feature Hedging: Correlated Features Break Narrow Sparse Autoencoders
David Chanin, Tomáš Dulka, Adrià Garriga-Alonso
It is assumed that sparse autoencoders (SAEs) decompose polysemantic activations into interpretable linear directions, as long as the activations are composed of sparse linear comb…
Open Problems in Mechanistic Interpretability
Lee Sharkey, Bilal Chughtai, Joshua Batson +26
Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks' capabilities in order to accomplish concrete scientific and engineering goa…
Investigating the Indirect Object Identification circuit in Mamba
Danielle Ensign, Adrià Garriga-Alonso
How well will current interpretability techniques generalize to future models? A relevant case study is Mamba, a recent recurrent architecture with scaling comparable to Transforme…
Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification
Thomas Kwa, Drake Thomas, Adrià Garriga-Alonso
When applying reinforcement learning from human feedback (RLHF), the reward is learned from data and, therefore, always has some error. It is common to mitigate this by regularizin…