8 citations · 17 across the 15 of their papers we have counts for
8 papers · 1 filter
Synthetic Persona Pretraining: Alignment from Token Zero
Julian Minder, Viktor Moskvoretskii, Raghav Singhal +12
As language-model-based AI is increasingly deployed in autonomous settings, aligning its goals and values with those of humans becomes critical. Today, alignment, and the assistant…
Mitigating Adaptive Attacks against Reasoning Models with Activation Consistency Training
Avidan Shah, Jannik Brinkmann, Rico Angell
As LLMs gain stronger reasoning capabilities, their extended chain-of-thought introduces new degrees of complexity for defending against adversarial jailbreaks and prompt injection…
Two Stages of Folding: Convergent Mechanisms in AI Protein Folding Trunks
Kevin Lu, Jannik Brinkmann, Stefan Huber +4
How do protein structure prediction models fold proteins? We investigate this question through causal interventions on the folding trunks of ESMFold, OpenFold, and Boltz-1. Across…
Jailbreak Transferability Emerges from Shared Representations
Rico Angell, Jannik Brinkmann, He He
Jailbreak transferability is the surprising phenomenon when an adversarial attack compromising one model also elicits harmful responses from other models. Despite widespread demons…
The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis
Aaron Mueller, Jannik Brinkmann, Millicent Li +10
Interpretability provides a toolset for understanding how and why neural networks behave in certain ways. However, there is little unity in the field: most studies employ ad-hoc ev…
Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models
Adam Karvonen, Benjamin Wright, Can Rager +6
What latent features are encoded in language model (LM) representations? Recent work on training sparse autoencoders (SAEs) to disentangle interpretable features in LM representati…
NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals
Jaden Fiotto-Kaufman, Alexander R. Loftus, Eric Todd +17
We introduce NNsight and NDIF, technologies that work in tandem to enable scientific study of the representations and computations learned by very large neural networks. NNsight is…
A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task
Jannik Brinkmann, Abhay Sheshadri, Victor Levoso +2
Transformers demonstrate impressive performance on a range of reasoning benchmarks. To evaluate the degree to which these abilities are a result of actual reasoning, existing work…