4.3k citations · 5.5k across the 13 of their papers we have counts for
15 papers
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
Evan Hubinger, Carson Denison, Jesse Mu +36
Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when giv…
Evaluating Language-Model Agents on Realistic Autonomous Tasks
Megan Kinniment, Lucas Jun Koba Sato, Haoxing Du +10
In this report, we explore the ability of language model agents to acquire resources, create copies of themselves, and adapt to novel challenges they encounter in the wild. We refe…
Model evaluation for extreme risks
Toby Shevlane, Sebastian Farquhar, Ben Garfinkel +18
Current approaches to building general-purpose AI systems tend to produce systems with both beneficial and harmful capabilities. Further progress in AI development could lead to ca…
Training language models to follow instructions with human feedback
Long Ouyang, Jeff Wu, Xu Jiang +17
Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic,…
Recursively Summarizing Books with Human Feedback
Jeff Wu, Long Ouyang, Daniel M. Ziegler +4
A major challenge for scaling machine learning is training models to perform tasks that are very difficult or time-consuming for humans to evaluate. We present progress on this pro…
Supervising strong learners by amplifying weak experts
Paul Christiano, Buck Shlegeris, Dario Amodei
Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution…
Unrestricted Adversarial Examples
Tom B. Brown, Nicholas Carlini, Chiyuan Zhang +3
We introduce a two-player contest for evaluating the safety and robustness of machine learning systems, with a large prize pool. Unlike most prior work in ML robustness, which stud…
AI safety via debate
Geoffrey Irving, Paul Christiano, Dario Amodei
To make AI systems broadly useful for challenging real-world tasks, we need them to learn complex human goals and preferences. One approach to specifying complex goals asks humans…
Deep reinforcement learning from human preferences
Paul Christiano, Jan Leike, Tom B. Brown +3
For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we exp…
A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
Chelsea Finn, Paul Christiano, Pieter Abbeel +1
Generative adversarial networks (GANs) are a recently proposed class of generative models in which a generator is trained to optimize a cost function that is being simultaneously l…
Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
Paul Christiano, Zain Shah, Igor Mordatch +5
Developing control policies in simulation is often more practical and safer than directly running experiments in the real world. This applies to policies obtained from planning and…
Concrete Problems in AI Safety
Dario Amodei, Chris Olah, Jacob Steinhardt +3
Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discus…
Theano: A Python framework for fast computation of mathematical expressions
The Theano Development Team, Rami Al-Rfou, Guillaume Alain +110
Theano is a Python library that allows to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. Since its introduction, it has bee…
Reflective Oracles: A Foundation for Classical Game Theory
Benja Fallenstein, Jessica Taylor, Paul F. Christiano
Classical game theory treats players as special---a description of a game contains a full, explicit enumeration of all players---even though in the real world, "players" are no mor…
Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic
Mihaly Barasz, Paul Christiano, Benja Fallenstein +3
We consider the one-shot Prisoner's Dilemma between algorithms with read-access to one anothers' source codes, and we use the modal logic of provability to build agents that can ac…