Learning from Suboptimal Demonstration via Self-Supervised Reward Regression

Abstract

Learning from Demonstration (LfD) seeks to democratize robotics by enabling non-roboticist end-users to teach robots to perform a task by providing a human demonstration. However, modern LfD techniques, e.g. inverse reinforcement learning (IRL), assume users provide at least stochastically optimal demonstrations. This assumption fails to hold in most real-world scenarios. Recent attempts to learn from sub-optimal demonstration leverage pairwise rankings and following the Luce-Shepard rule. However, we show these approaches make incorrect assumptions and thus suffer from brittle, degraded performance. We overcome these limitations in developing a novel approach that bootstraps off suboptimal demonstrations to synthesize optimality-parameterized data to train an idealized reward function. We empirically validate we learn an idealized reward function with ~0.95 correlation with ground-truth reward versus ~0.75 for prior work. We can then train policies achieving ~200% improvement over the suboptimal demonstration and ~90% improvement over prior work. We present a physical demonstration of teaching a robot a topspin strike in table tennis that achieves 32% faster returns and 40% more topspin than user demonstration.

In Proceedings of the Conference on Robot Learning (CoRL '20)

References in corpus (3)

• Soft Actor-Critic Algorithms and Applications
• DART: Noise Injection for Robust Imitation Learning
• Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations

Cited by in corpus (9)

• Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality
• Robotic Table Tennis: A Case Study into a High Speed Learning System
• The Effect of Robot Skill Level and Communication in Rapid, Proximate Human-Robot Collaboration
• Semi-Supervised Imitation Learning of Team Policies from Suboptimal Demonstrations
• Neural Networks with Inputs Based on Domain of Dependence and A Converging Sequence for Solving Conservation Laws, Part I: 1D Riemann Problems
• Marginal MAP Estimation for Inverse RL under Occlusion with Observer Noise
• Towards Sample-efficient Apprenticeship Learning from Suboptimal Demonstration
• Back to Reality for Imitation Learning
• FireCommander: An Interactive, Probabilistic Multi-agent Environment for Heterogeneous Robot Teams