Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations

Abstract

The performance of imitation learning is typically upper-bounded by the performance of the demonstrator. While recent empirical results demonstrate that ranked demonstrations allow for better-than-demonstrator performance, preferences over demonstrations may be difficult to obtain, and little is known theoretically about when such methods can be expected to successfully extrapolate beyond the performance of the demonstrator. To address these issues, we first contribute a sufficient condition for better-than-demonstrator imitation learning and provide theoretical results showing why preferences over demonstrations can better reduce reward function ambiguity when performing inverse reinforcement learning. Building on this theory, we introduce Disturbance-based Reward Extrapolation (D-REX), a ranking-based imitation learning method that injects noise into a policy learned through behavioral cloning to automatically generate ranked demonstrations. These ranked demonstrations are used to efficiently learn a reward function that can then be optimized using reinforcement learning. We empirically validate our approach on simulated robot and Atari imitation learning benchmarks and show that D-REX outperforms standard imitation learning approaches and can significantly surpass the performance of the demonstrator. D-REX is the first imitation learning approach to achieve significant extrapolation beyond the demonstrator's performance without additional side-information or supervision, such as rewards or human preferences. By generating rankings automatically, we show that preference-based inverse reinforcement learning can be applied in traditional imitation learning settings where only unlabeled demonstrations are available.

In proceedings of 3rd Conference on Robot Learning (CoRL) 2019

References in corpus (12)

• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Deep Q-learning from Demonstrations
• Maximum Entropy Deep Inverse Reinforcement Learning
• Visualizing and Understanding Atari Agents
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Reinforcement Learning from Imperfect Demonstrations
• DART: Noise Injection for Robust Imitation Learning
• Occam's razor is insufficient to infer the preferences of irrational agents
• Safe end-to-end imitation learning for model predictive control
• Imitation Learning from Imperfect Demonstration
• Towards Resolving Unidentifiability in Inverse Reinforcement Learning
• Inverse Reinforcement Learning with Multiple Ranked Experts

Cited by in corpus (6)

• Learning from Suboptimal Demonstration via Self-Supervised Reward Regression
• Imitation Learning: Progress, Taxonomies and Challenges
• Learning Reward Functions from Scale Feedback
• Inverse Reinforcement Learning via Matching of Optimality Profiles
• Towards Sample-efficient Apprenticeship Learning from Suboptimal Demonstration
• Teachable Reinforcement Learning via Advice Distillation