24 citations · 39 across the 12 of their papers we have counts for
6 papers · 1 filter
School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs
Mia Taylor, James Chua, Jan Betley +2
Reward hacking--where agents exploit flaws in imperfect reward functions rather than performing tasks as intended--poses risks for AI alignment. Reward hacking has been observed in…
Auditing language models for hidden objectives
Samuel Marks, Johannes Treutlein, Trenton Bricken +32
We study the feasibility of conducting alignment audits: investigations into whether models have undesired objectives. As a testbed, we train a language model with a hidden objecti…
Alignment faking in large language models
Ryan Greenblatt, Carson Denison, Benjamin Wright +17
We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its beha…
Incentivizing honest performative predictions with proper scoring rules
Caspar Oesterheld, Johannes Treutlein, Emery Cooper +1
Proper scoring rules incentivize experts to accurately report beliefs, assuming predictions cannot influence outcomes. We relax this assumption and investigate incentives when pred…
Conditioning Predictive Models: Risks and Strategies
Evan Hubinger, Adam Jermyn, Johannes Treutlein +2
Our intention is to provide a definitive reference on what it would take to safely make use of generative/predictive models in the absence of a solution to the Eliciting Latent Kno…
A New Formalism, Method and Open Issues for Zero-Shot Coordination
Johannes Treutlein, Michael Dennis, Caspar Oesterheld +1
In many coordination problems, independently reasoning humans are able to discover mutually compatible policies. In contrast, independently trained self-play policies are often mut…