24 citations · 39 across the 12 of their papers we have counts for
12 papers
Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Values
Jan Betley, Johannes Treutlein, Jan Dubiński +5
People use language models for practical questions whose answers are difficult to verify. We show that models exhibit covert value leakage: the information they provide is influenc…
School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs
Mia Taylor, James Chua, Jan Betley +2
Reward hacking--where agents exploit flaws in imperfect reward functions rather than performing tasks as intended--poses risks for AI alignment. Reward hacking has been observed in…
Auditing language models for hidden objectives
Samuel Marks, Johannes Treutlein, Trenton Bricken +32
We study the feasibility of conducting alignment audits: investigations into whether models have undesired objectives. As a testbed, we train a language model with a hidden objecti…
Alignment faking in large language models
Ryan Greenblatt, Carson Denison, Benjamin Wright +17
We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its beha…
Modeling evidential cooperation in large worlds
Johannes Treutlein
Evidential cooperation in large worlds (ECL) refers to the idea that humans and other agents can benefit by cooperating with similar agents with differing values in causally discon…
Incentivizing honest performative predictions with proper scoring rules
Caspar Oesterheld, Johannes Treutlein, Emery Cooper +1
Proper scoring rules incentivize experts to accurately report beliefs, assuming predictions cannot influence outcomes. We relax this assumption and investigate incentives when pred…
Conditioning Predictive Models: Risks and Strategies
Evan Hubinger, Adam Jermyn, Johannes Treutlein +2
Our intention is to provide a definitive reference on what it would take to safely make use of generative/predictive models in the absence of a solution to the Eliciting Latent Kno…
Path Independent Equilibrium Models Can Better Exploit Test-Time Computation
Cem Anil, Ashwini Pokle, Kaiqu Liang +5
Designing networks capable of attaining better performance with an increased inference budget is important to facilitate generalization to harder problem instances. Recent efforts…
Similarity-based cooperative equilibrium
Caspar Oesterheld, Johannes Treutlein, Roger Grosse +2
As machine learning agents act more autonomously in the world, they will increasingly interact with each other. Unfortunately, in many social dilemmas like the one-shot Prisoner's…
COLA: Consistent Learning with Opponent-Learning Awareness
Timon Willi, Alistair Letcher, Johannes Treutlein +1
Learning in general-sum games is unstable and frequently leads to socially undesirable (Pareto-dominated) outcomes. To mitigate this, Learning with Opponent-Learning Awareness (LOL…
Normative Disagreement as a Challenge for Cooperative AI
Julian Stastny, Maxime Riché, Alexander Lyzhov +3
Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. Ho…
A New Formalism, Method and Open Issues for Zero-Shot Coordination
Johannes Treutlein, Michael Dennis, Caspar Oesterheld +1
In many coordination problems, independently reasoning humans are able to discover mutually compatible policies. In contrast, independently trained self-play policies are often mut…