Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
arXiv:2108.05382
Abstract
A promising approach to solving challenging long-horizon tasks has been to extract behavior priors (skills) by fitting generative models to large offline datasets of demonstrations. However, such generative models inherit the biases of the underlying data and result in poor and unusable skills when trained on imperfect demonstration data. To better align skill extraction with human intent we present Skill Preferences (SkiP), an algorithm that learns a model over human preferences and uses it to extract human-aligned skills from offline data. After extracting human-preferred skills, SkiP also utilizes human feedback to solve down-stream tasks with RL. We show that SkiP enables a simulated kitchen robot to solve complex multi-step manipulation tasks and substantially outperforms prior leading RL algorithms with human preferences as well as leading skill extraction algorithms without human preferences.
8 pages,6 figures. for associated code and video, see http://sites.google.com/view/skill-pref
References in corpus (13)
- Solving Rubik's Cube with a Robot Hand
- Deep reinforcement learning from human preferences
- Fine-Tuning Language Models from Human Preferences
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Behavior Regularized Offline Reinforcement Learning
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- Scalable agent alignment via reward modeling: a research direction
- Interactive Learning from Policy-Dependent Human Feedback
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
- Deep Reinforcement Learning from Policy-Dependent Human Feedback
- PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
- Avoiding Side Effects in Complex Environments