Primal Wasserstein Imitation Learning

Abstract

Imitation Learning (IL) methods seek to match the behavior of an agent with that of an expert. In the present work, we propose a new IL method based on a conceptually simple algorithm: Primal Wasserstein Imitation Learning (PWIL), which ties to the primal form of the Wasserstein distance between the expert and the agent state-action distributions. We present a reward function which is derived offline, as opposed to recent adversarial IL algorithms that learn a reward function through interactions with the environment, and which requires little fine-tuning. We show that we can recover expert behavior on a variety of continuous control tasks of the MuJoCo domain in a sample efficient manner in terms of agent interactions and of expert interactions with the environment. Finally, we show that the behavior of the agent we train matches the behavior of the expert with the Wasserstein distance, rather than the commonly used proxy of performance.

arXiv commentPublished in International Conference on Learning Representations (ICLR 2021)

References in corpus (16)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Distributed Distributional Deterministic Policy Gradients
• Deep Q-learning from Demonstrations
• Acme: A Research Framework for Distributed Reinforcement Learning
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• Imitating Latent Policies from Observation
• Wasserstein Adversarial Imitation Learning
• Imitation Learning as $f$-Divergence Minimization
• State Alignment-based Imitation Learning
• Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation
• Risk-Sensitive Generative Adversarial Imitation Learning

Cited by in corpus (14)

• Strictly Batch Imitation Learning by Energy-based Distribution Matching
• Offline Reinforcement Learning with Pseudometric Learning
• Adversarial Intrinsic Motivation for Reinforcement Learning
• Inverse Decision Modeling: Learning Interpretable Representations of Behavior
• Cross-Domain Imitation Learning via Optimal Transport
• Imitation Learning with Sinkhorn Distances
• Regularized Inverse Reinforcement Learning
• Sliced Multi-Marginal Optimal Transport
• Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularization
• Learning Synthetic to Real Transfer for Localization and Navigational Tasks
• Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration
• Distributionally-Constrained Policy Optimization via Unbalanced Optimal Transport
• Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap
• Wasserstein Distance Maximizing Intrinsic Control