Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification

Abstract

Reinforcement learning (RL) algorithms assume that users specify tasks by manually writing down a reward function. However, this process can be laborious and demands considerable technical expertise. Can we devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes? In this paper, we derive a control algorithm that maximizes the future probability of these successful outcome examples. Prior work has approached similar problems with a two-stage process, first learning a reward function and then optimizing this reward function using another RL algorithm. In contrast, our method directly learns a value function from transitions and successful outcomes, without learning this intermediate reward function. Our method therefore requires fewer hyperparameters to tune and lines of code to debug. We show that our method satisfies a new data-driven Bellman equation, where examples take the place of the typical reward function term. Experiments show that our approach outperforms prior methods that learn explicit reward functions.

arXiv commentNeurIPS 2021 (oral). Website with code, videos, and blog post: https://ben-eysenbach.github.io/rce

References in corpus (18)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Maximum Entropy Deep Inverse Reinforcement Learning
• The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
• Deep Successor Reinforcement Learning
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition
• Few-Shot Goal Inference for Visuomotor Learning and Planning
• End-to-End Robotic Reinforcement Learning without Reward Engineering
• Imitation Learning via Off-Policy Distribution Matching
• Off-Policy Evaluation via Off-Policy Classification
• Task-Relevant Adversarial Imitation Learning
• Offline Learning from Demonstrations and Unlabeled Experience
• Positive-Unlabeled Reward Learning
• C-Learning: Learning to Achieve Goals via Recursive Classification
• Semi-supervised reward learning for offline reinforcement learning

Cited by in corpus (4)

• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
• Visual Adversarial Imitation Learning using Variational Models
• Adversarial Intrinsic Motivation for Reinforcement Learning
• Offline Preference-Based Apprenticeship Learning