Wasserstein Adversarial Imitation Learning

Abstract

Imitation Learning describes the problem of recovering an expert policy from demonstrations. While inverse reinforcement learning approaches are known to be very sample-efficient in terms of expert demonstrations, they usually require problem-dependent reward functions or a (task-)specific reward-function regularization. In this paper, we show a natural connection between inverse reinforcement learning approaches and Optimal Transport, that enables more general reward functions with desirable properties (e.g., smoothness). Based on our observation, we propose a novel approach called Wasserstein Adversarial Imitation Learning. Our approach considers the Kantorovich potentials as a reward function and further leverages regularized optimal transport to enable large-scale applications. In several robotic experiments, our approach outperforms the baselines in terms of average cumulative rewards and shows a significant improvement in sample-efficiency, by requiring just one expert demonstration.

References in corpus (3)

• Trust Region Policy Optimization
• Smooth and Sparse Optimal Transport
• Learning to Optimize via Wasserstein Deep Inverse Optimal Control

Cited by in corpus (15)

• Modeling Human Driving Behavior through Generative Adversarial Imitation Learning
• Primal Wasserstein Imitation Learning
• State Alignment-based Imitation Learning
• An Imitation from Observation Approach to Transfer Learning with Dynamics Mismatch
• Strictly Batch Imitation Learning by Energy-based Distribution Matching
• Decision Making for Autonomous Driving via Augmented Adversarial Inverse Reinforcement Learning
• Non-Adversarial Imitation Learning and its Connections to Adversarial Methods
• Robust Imitation Learning from Noisy Demonstrations
• Adversarial Intrinsic Motivation for Reinforcement Learning
• Imitation Learning with Sinkhorn Distances
• Learning from Imperfect Demonstrations from Agents with Varying Dynamics
• Reward is enough for convex MDPs
• Inverse Reinforcement Learning via Matching of Optimality Profiles
• Wasserstein Distance Maximizing Intrinsic Control
• Distributionally-Constrained Policy Optimization via Unbalanced Optimal Transport