Policy Search by Target Distribution Learning for Continuous Control
arXiv:1905.11041
Abstract
We observe that several existing policy gradient methods (such as vanilla policy gradient, PPO, A2C) may suffer from overly large gradients when the current policy is close to deterministic (even in some very simple environments), leading to an unstable training process. To address this issue, we propose a new method, called \emph{target distribution learning} (TDL), for policy improvement in reinforcement learning. TDL alternates between proposing a target distribution and training the policy network to approach the target distribution. TDL is more effective in constraining the KL divergence between updated policies, and hence leads to more stable policy improvements over iterations. Our experiments show that TDL algorithms perform comparably to (or better than) state-of-the-art algorithms for most continuous control tasks in the MuJoCo environment while being more stable in training.
arXiv commentAAAI-20 (oral)
References in corpus (15)
- Continuous control with deep reinforcement learning
- Trust Region Policy Optimization
- Prioritized Experience Replay
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
- Continuous Deep Q-Learning with Model-based Acceleration
- Implicit Quantile Networks for Distributional Reinforcement Learning
- Maximum a Posteriori Policy Optimisation
- Sample Efficient Actor-Critic with Experience Replay
- Simple random search provides a competitive approach to reinforcement learning
- On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
- Relative Entropy Regularized Policy Iteration
- Potential theory, path integrals and the Laplacian of the indicator
- Remember and Forget for Experience Replay