Learning Human Objectives by Evaluating Hypothetical Behavior

Abstract

We seek to align agent behavior with a user's objectives in a reinforcement learning setting with unknown dynamics, an unknown reward function, and unknown unsafe states. The user knows the rewards and unsafe states, but querying the user is expensive. To address this challenge, we propose an algorithm that safely and interactively learns a model of the user's reward function. We start with a generative model of initial states and a forward dynamics model trained on off-policy data. Our method uses these models to synthesize hypothetical behaviors, asks the user to label the behaviors with rewards, and trains a neural network to predict the rewards. The key idea is to actively synthesize the hypothetical behaviors from scratch by maximizing tractable proxies for the value of information, without interacting with the environment. We call this method reward query synthesis via trajectory optimization (ReQueST). We evaluate ReQueST with simulated users on a state-based 2D navigation task and the image-based Car Racing video game. The results show that ReQueST significantly outperforms prior methods in learning reward models that transfer to new environments with different initial state distributions. Moreover, ReQueST safely trains the reward model to detect unsafe states, and corrects reward hacking before deploying the agent.

arXiv commentPublished at International Conference on Machine Learning (ICML) 2020

References in corpus (12)

• Auto-Encoding Variational Bayes
• Bayesian Active Learning for Classification and Preference Learning
• Recurrent World Models Facilitate Policy Evolution
• Model-Based Reinforcement Learning for Atari
• Learning Latent Dynamics for Planning from Pixels
• Safe Exploration in Continuous Action Spaces
• Scalable agent alignment via reward modeling: a research direction
• Interactive Learning from Policy-Dependent Human Feedback
• Supervising strong learners by amplifying weak experts
• The EMPATHIC Framework for Task Learning from Implicit Human Feedback
• Accelerating Reinforcement Learning Agent with EEG-based Implicit Human Feedback
• Active Inverse Reward Design

Cited by in corpus (4)

• Feature Expansive Reward Learning: Rethinking Human Input
• Negative Human Rights as a Basis for Long-term AI Safety and Regulation
• Reinforcement Learning Under Moral Uncertainty
• Avoiding Tampering Incentives in Deep RL via Decoupled Approval