Wasserstein Adversarial Imitation Learning

Abstract

Imitation Learning describes the problem of recovering an expert policy from demonstrations. While inverse reinforcement learning approaches are known to be very sample-efficient in terms of expert demonstrations, they usually require problem-dependent reward functions or a (task-)specific reward-function regularization. In this paper, we show a natural connection between inverse reinforcement learning approaches and Optimal Transport, that enables more general reward functions with desirable properties (e.g., smoothness). Based on our observation, we propose a novel approach called Wasserstein Adversarial Imitation Learning. Our approach considers the Kantorovich potentials as a reward function and further leverages regularized optimal transport to enable large-scale applications. In several robotic experiments, our approach outperforms the baselines in terms of average cumulative rewards and shows a significant improvement in sample-efficiency, by requiring just one expert demonstration.

Cited by in corpus (6)

• State Alignment-based Imitation Learning
• Non-Adversarial Imitation Learning and its Connections to Adversarial Methods
• Learning from Imperfect Demonstrations from Agents with Varying Dynamics
• Inverse Reinforcement Learning via Matching of Optimality Profiles
• Distributionally-Constrained Policy Optimization via Unbalanced Optimal Transport
• Wasserstein Distance Maximizing Intrinsic Control