Offline Preference-Based Apprenticeship Learning
arXiv:2107.09251
Abstract
Learning a reward function from human preferences is challenging as it typically requires having a high-fidelity simulator or using expensive and potentially unsafe actual physical rollouts in the environment. However, in many tasks the agent might have access to offline data from related tasks in the same target environment. While offline data is increasingly being used to aid policy optimization via offline RL, our observation is that it can be a surprisingly rich source of information for preference learning as well. We propose an approach that uses an offline dataset to craft preference queries via pool-based active learning, learns a distribution over reward functions, and optimizes a corresponding policy via offline RL. Crucially, our proposed approach does not require actual physical rollouts or an accurate simulator for either the reward learning or policy optimization steps. To test our approach, we identify a subset of existing offline RL benchmarks that are well suited for offline reward learning and also propose new offline apprenticeship learning benchmarks which allow for more open-ended behaviors. Our empirical results suggest that combining offline RL with learned human preferences can enable an agent to learn to perform novel tasks that were not explicitly shown in the offline data.
arXiv commentICML Workshop on Human-AI Collaboration in Sequential Decision-Making, 2021
References in corpus (15)
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
- Conservative Q-Learning for Offline Reinforcement Learning
- Deep reinforcement learning from human preferences
- Bayesian Active Learning for Classification and Preference Learning
- Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- Query-Efficient Imitation Learning for End-to-End Autonomous Driving
- Scaling data-driven robotics with reward sketching and batch reinforcement learning
- Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
- Offline Learning from Demonstrations and Unlabeled Experience
- PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
- Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
- DERAIL: Diagnostic Environments for Reward And Imitation Learning