Learning Policies through Quantile Regression
arXiv:1906.11941
Abstract
Policy gradient based reinforcement learning algorithms coupled with neural networks have shown success in learning complex policies in the model free continuous action space control setting. However, explicitly parameterized policies are limited by the scope of the chosen parametric probability distribution. We show that alternatively to the likelihood based policy gradient, a related objective can be optimized through advantage weighted quantile regression. Our approach models the policy implicitly in the network, which gives the agent the freedom to approximate any distribution in each action dimension, not limiting its capabilities to the commonly used unimodal Gaussian parameterization. This broader spectrum of policies makes our algorithm suitable for problems where Gaussian policies cannot fit the optimal policy. Moreover, our results on the MuJoCo physics simulator benchmarks are comparable or superior to state-of-the-art on-policy methods.
arXiv commentPreprint
References in corpus (16)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Soft Actor-Critic Algorithms and Applications
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
- Reinforcement Learning with Deep Energy-Based Policies
- Deep Reinforcement Learning that Matters
- Implicit Quantile Networks for Distributional Reinforcement Learning
- Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
- Equivalence Between Policy Gradients and Soft Q-Learning
- Distributional Reinforcement Learning with Quantile Regression
- Autoregressive Quantile Networks for Generative Modeling
- VIREL: A Variational Inference Framework for Reinforcement Learning
- Policy Optimization as Wasserstein Gradient Flows
- On Wasserstein Reinforcement Learning and the Fokker-Planck equation