Offline Learning from Demonstrations and Unlabeled Experience

Abstract

Behavior cloning (BC) is often practical for robot learning because it allows a policy to be trained offline without rewards, by supervised learning on expert demonstrations. However, BC does not effectively leverage what we will refer to as unlabeled experience: data of mixed and unknown quality without reward annotations. This unlabeled data can be generated by a variety of sources such as human teleoperation, scripted policies and other agents on the same robot. Towards data-driven offline robot learning that can use this unlabeled experience, we introduce Offline Reinforced Imitation Learning (ORIL). ORIL first learns a reward function by contrasting observations from demonstrator and unlabeled trajectories, then annotates all data with the learned reward, and finally trains an agent via offline reinforcement learning. Across a diverse set of continuous control and simulated robotic manipulation tasks, we show that ORIL consistently outperforms comparable BC agents by effectively leveraging unlabeled experience.

Accepted to Offline Reinforcement Learning Workshop at Neural Information Processing Systems (2020)

References in corpus (9)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Learning human behaviors from motion capture by adversarial imitation
• Critic Regularized Regression
• Observe and Look Further: Achieving Consistent Performance on Atari
• Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
• Reinforced Imitation in Heterogeneous Action Space

Cited by in corpus (8)

• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Semi-supervised reward learning for offline reinforcement learning
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
• Offline Preference-Based Apprenticeship Learning
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• Offline Inverse Reinforcement Learning