Regularly Updated Deterministic Policy Gradient Algorithm
arXiv:2007.00169
Abstract
Deep Deterministic Policy Gradient (DDPG) algorithm is one of the most well-known reinforcement learning methods. However, this method is inefficient and unstable in practical applications. On the other hand, the bias and variance of the Q estimation in the target function are sometimes difficult to control. This paper proposes a Regularly Updated Deterministic (RUD) policy gradient algorithm for these problems. This paper theoretically proves that the learning procedure with RUD can make better use of new data in replay buffer than the traditional procedure. In addition, the low variance of the Q value in RUD is more suitable for the current Clipped Double Q-learning strategy. This paper has designed a comparison experiment against previous methods, an ablation experiment with the original DDPG, and other analytical experiments in Mujoco environments. The experimental results demonstrate the effectiveness and superiority of RUD.
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Addressing Function Approximation Error in Actor-Critic Methods
- Distributed Prioritized Experience Replay
- Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
- Count-Based Exploration with Neural Density Models
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
- Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
- CEM-RL: Combining evolutionary and gradient-based methods for policy search
- Evolution-Guided Policy Gradient in Reinforcement Learning
- Recruitment-imitation Mechanism for Evolutionary Reinforcement Learning