Efficient Wasserstein Natural Gradients for Reinforcement Learning

Abstract

A novel optimization approach is proposed for application to policy gradient methods and evolution strategies for reinforcement learning (RL). The procedure uses a computationally efficient Wasserstein natural gradient (WNG) descent that takes advantage of the geometry induced by a Wasserstein penalty to speed optimization. This method follows the recent theme in RL of including a divergence penalty in the objective to establish a trust region. Experiments on challenging tasks demonstrate improvements in both computational cost and performance over advanced baselines.

References in corpus (8)

• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Stochastic Optimization for Large-scale Optimal Transport
• Transport information geometry I: Riemannian calculus on probability simplex
• Ricci curvature for parametric statistics via optimal transport
• Optimal transport natural gradient for statistical manifolds with continuous sample space
• Implicit Policy for Reinforcement Learning
• Learning to Score Behaviors for Guided Policy Optimization
• First-Order Preconditioning via Hypergradient Descent