ADER:Adapting between Exploration and Robustness for Actor-Critic Methods

Abstract

Combining off-policy reinforcement learning methods with function approximators such as neural networks has been found to lead to overestimation of the value function and sub-optimal solutions. Improvement such as TD3 has been proposed to address this issue. However, we surprisingly find that its performance lags behind the vanilla actor-critic methods (such as DDPG) in some primitive environments. In this paper, we show that the failure of some cases can be attributed to insufficient exploration. We reveal the culprit of insufficient exploration in TD3, and propose a novel algorithm toward this problem that ADapts between Exploration and Robustness, namely ADER. To enhance the exploration ability while eliminating the overestimation bias, we introduce a dynamic penalty term in value estimation calculated from estimated uncertainty, which takes into account different compositions of the uncertainty in different learning stages. Experiments in several challenging environments demonstrate the supremacy of the proposed method in continuous control tasks.

References in corpus (8)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• Better Exploration with Optimistic Actor-Critic
• Efficient exploration with Double Uncertain Value Networks

Cited by in corpus (1)

• Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control