Semi-supervised reward learning for offline reinforcement learning
arXiv:2012.06899
Abstract
In offline reinforcement learning (RL) agents are trained using a logged dataset. It appears to be the most natural route to attack real-life applications because in domains such as healthcare and robotics interactions with the environment are either expensive or unethical. Training agents usually requires reward functions, but unfortunately, rewards are seldom available in practice and their engineering is challenging and laborious. To overcome this, we investigate reward learning under the constraint of minimizing human reward annotations. We consider two types of supervision: timestep annotations and demonstrations. We propose semi-supervised learning algorithms that learn from limited annotations and incorporate unlabelled data. In our experiments with a simulated robotic arm, we greatly improve upon behavioural cloning and closely approach the performance achieved with ground truth rewards. We further investigate the relationship between the quality of the reward model and the final policies. We notice, for example, that the reward models do not need to be perfect to result in useful policies.
arXiv commentAccepted to Offline Reinforcement Learning Workshop at Neural Information Processing Systems (2020)
References in corpus (10)
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Weakly Supervised Object Localization with Multi-fold Multiple Instance Learning
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- Learning human behaviors from motion capture by adversarial imitation
- Acme: A Research Framework for Distributed Reinforcement Learning
- BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
- End-to-End Robotic Reinforcement Learning without Reward Engineering
- Task-Relevant Adversarial Imitation Learning
- Offline Learning from Demonstrations and Unlabeled Experience
- Positive-Unlabeled Reward Learning