On the Estimation Bias in Double Q-Learning
arXiv:2109.14419
Abstract
Double Q-learning is a classical method for reducing overestimation bias, which is caused by taking maximum estimated values in the Bellman operation. Its variants in the deep Q-learning paradigm have shown great promise in producing reliable value prediction and improving learning performance. However, as shown by prior work, double Q-learning is not fully unbiased and suffers from underestimation bias. In this paper, we show that such underestimation bias may lead to multiple non-optimal fixed points under an approximate Bellman operator. To address the concerns of converging to non-optimal stationary solutions, we propose a simple but effective approach as a partial fix for the underestimation bias in double Q-learning. This approach leverages an approximate dynamic programming to bound the target value. We extensively evaluate our proposed method in the Atari benchmark tasks and demonstrate its significant improvement over baseline algorithms.
arXiv commentThirty-Fifth Conference on Neural Information Processing Systems (NeurIPS 2021)
References in corpus (17)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Dueling Network Architectures for Deep Reinforcement Learning
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Dopamine: A Research Framework for Deep Reinforcement Learning
- Model-Free Episodic Control
- Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
- Deep Reinforcement Learning and the Deadly Triad
- Neural Episodic Control
- Towards Characterizing Divergence in Deep Q-Learning
- Maxmin Q-learning: Controlling the Estimation Bias of Q-learning
- Better Exploration with Optimistic Actor-Critic
- Diagnosing Bottlenecks in Deep Q-learning Algorithms
- Revisiting the Softmax Bellman Operator: New Benefits and New Perspective
- Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
- Generalizable Episodic Memory for Deep Reinforcement Learning