Reward learning from human preferences and demonstrations in Atari
arXiv:1811.06521
Abstract
To solve complex real-world problems with reinforcement learning, we cannot rely on manually specified reward functions. Instead, we can have humans communicate an objective to the agent directly. In this work, we combine two approaches to learning from human feedback: expert demonstrations and trajectory preferences. We train a deep neural network to model the reward function and use its predicted reward to train an DQN-based deep reinforcement learning agent on 9 Atari games. Our approach beats the imitation learning baseline in 7 games and achieves strictly superhuman performance on 2 games without using game rewards. Additionally, we investigate the goodness of fit of the reward model, present some reward hacking problems, and study the effects of noise in the human labels.
arXiv commentNIPS 2018
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Explaining and Harnessing Adversarial Examples
- Playing Atari with Deep Reinforcement Learning
- Prioritized Experience Replay
- Dueling Network Architectures for Deep Reinforcement Learning
- Deep reinforcement learning from human preferences
- Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
- Generative Adversarial Imitation Learning
- Variational Intrinsic Control
- Reinforcement and Imitation Learning for Diverse Visuomotor Skills
- Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrations
- Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worlds
- Actor-Critic Reinforcement Learning with Simultaneous Human Control and Feedback