Positive-Unlabeled Reward Learning

Abstract

Learning reward functions from data is a promising path towards achieving scalable Reinforcement Learning (RL) for robotics. However, a major challenge in training agents from learned reward models is that the agent can learn to exploit errors in the reward model to achieve high reward behaviors that do not correspond to the intended task. These reward delusions can lead to unintended and even dangerous behaviors. On the other hand, adversarial imitation learning frameworks tend to suffer the opposite problem, where the discriminator learns to trivially distinguish agent and expert behavior, resulting in reward models that produce low reward signal regardless of the input state. In this paper, we connect these two classes of reward learning methods to positive-unlabeled (PU) learning, and we show that by applying a large-scale PU learning algorithm to the reward learning problem, we can address both the reward under- and over-estimation problems simultaneously. Our approach drastically improves both GAIL and supervised reward learning, without any additional assumptions.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• DeepMind Control Suite
• Distributed Prioritized Experience Replay
• Distributed Distributional Deterministic Policy Gradients
• Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
• Theoretical Comparisons of Positive-Unlabeled Learning against Positive-Negative Learning
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• Sample-Efficient Imitation Learning via Generative Adversarial Nets
• Task-Relevant Adversarial Imitation Learning
• Reinforced Imitation in Heterogeneous Action Space

Cited by in corpus (7)

• What Matters for Adversarial Imitation Learning?
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Human-in-the-Loop Imitation Learning using Remote Teleoperation
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Semi-supervised reward learning for offline reinforcement learning
• Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification