Learning to Score Behaviors for Guided Policy Optimization
arXiv:1906.04349
Abstract
We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space. We show that by utilizing the dual formulation of the WD, we can learn score functions over policy behaviors that can in turn be used to lead policy optimization towards (or away from) (un)desired behaviors. Combined with smoothed WDs, the dual formulation allows us to devise efficient algorithms that take stochastic gradient descent steps through WD regularizers. We incorporate these regularizers into two novel on-policy algorithms, Behavior-Guided Policy Gradient and Behavior-Guided Evolution Strategies, which we demonstrate can outperform existing methods in a variety of challenging environments. We also provide an open source demo.
References in corpus (12)
- Auto-Encoding Variational Bayes
- Playing Atari with Deep Reinforcement Learning
- Trust Region Policy Optimization
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Noisy Networks for Exploration
- Implicit Quantile Networks for Distributional Reinforcement Learning
- An Analysis of Categorical Distributional Reinforcement Learning
- Statistics and Samples in Distributional Reinforcement Learning
- Wasserstein Fair Classification
- On Wasserstein Reinforcement Learning and the Fokker-Planck equation
- Nonlinear Distributional Gradient Temporal-Difference Learning
- Distributional reinforcement learning with linear function approximation