91 citations · 189 across the 19 of their papers we have counts for
13 papers · 1 filter
Explaining grokking through circuit efficiency
Vikrant Varma, Rohin Shah, Zachary Kenton +2
One of the most surprising puzzles in neural network generalisation is grokking: a network with perfect training accuracy but poor generalisation will, upon further training, trans…
Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla
Tom Lieberum, Matthew Rahtz, János Kramár +4
\emph{Circuit analysis} is a promising technique for understanding the internal mechanisms of language models. However, existing analyses are done in small models far from the stat…
Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals
Rohin Shah, Vikrant Varma, Ramana Kumar +4
The field of AI alignment is concerned with AI systems that pursue unintended goals. One commonly studied mechanism by which an unintended goal might arise is specification gaming,…
An Empirical Investigation of Representation Learning for Imitation
Xin Chen, Sam Toyer, Cody Wild +9
Imitation learning often needs a large demonstration set in order to handle the full range of situations that an agent might find itself in during deployment. However, collecting e…
The MineRL BASALT Competition on Learning from Human Feedback
Rohin Shah, Cody Wild, Steven H. Wang +10
The last decade has seen a significant increase of interest in deep learning research, with many public successes that have demonstrated its potential. As such, these systems are n…
Learning What To Do by Simulating the Past
David Lindner, Rohin Shah, Pieter Abbeel +1
Since reward functions are hard to specify, recent work has focused on learning policies from human feedback. However, such approaches are impeded by the expense of acquiring such…
Combining Reward Information from Multiple Sources
Dmitrii Krasheninnikov, Rohin Shah, Herke van Hoof
Given two sources of evidence about a latent variable, one can combine the information from both by multiplying the likelihoods of each piece of evidence. However, when one or both…
Evaluating the Robustness of Collaborative Agents
Paul Knott, Micah Carroll, Sam Devlin +4
In order for agents trained by deep reinforcement learning to work alongside humans in realistic settings, we will need to ensure that the agents are \emph{robust}. Since the real…
The MAGICAL Benchmark for Robust Imitation
Sam Toyer, Rohin Shah, Andrew Critch +1
Imitation Learning (IL) algorithms are typically evaluated in the same environment that was used to create demonstrations. This rewards precise reproduction of demonstrations in on…
On the Utility of Learning about Humans for Human-AI Coordination
Micah Carroll, Rohin Shah, Mark K. Ho +4
While we would like agents that can coordinate with humans, current algorithms such as self-play and population-based training create agents that can coordinate with themselves. Ag…
On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference
Rohin Shah, Noah Gundotra, Pieter Abbeel +1
Our goal is for agents to optimize the right reward function, despite how difficult it is for us to specify what that is. Inverse Reinforcement Learning (IRL) enables us to infer r…
Preferences Implicit in the State of the World
Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander +2
Reinforcement learning (RL) agents optimize only the features specified in a reward function and are indifferent to anything left out inadvertently. This means that we must not onl…
Active Inverse Reward Design
Sören Mindermann, Rohin Shah, Adam Gleave +1
Designers of AI agents often iterate on the reward function in a trial-and-error process until they get the desired behavior, but this only guarantees good behavior in the training…