Convergent and Efficient Deep Q Network Algorithm
arXiv:2106.15419
Abstract
Despite the empirical success of the deep Q network (DQN) reinforcement learning algorithm and its variants, DQN is still not well understood and it does not guarantee convergence. In this work, we show that DQN can indeed diverge and cease to operate in realistic settings. Although there exist gradient-based convergent methods, we show that they actually have inherent problems in learning dynamics which cause them to fail even in simple tasks. To overcome these problems, we propose a convergent DQN algorithm (C-DQN) that is guaranteed to converge and can work with large discount factors (0.9998). It learns robustly in difficult settings and can learn several difficult games in the Atari 2600 benchmark that DQN fails to solve. Our codes have been publicly released and can be used to reproduce our results.
References in corpus (19)
- Continuous control with deep reinforcement learning
- Dueling Network Architectures for Deep Reinforcement Learning
- Dota 2 with Large Scale Deep Reinforcement Learning
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Reinforcement Learning with Deep Energy-Based Policies
- Agent57: Outperforming the Atari Human Benchmark
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
- Deep Reinforcement Learning and the Deadly Triad
- Reinforcement Learning from Imperfect Demonstrations
- Observe and Look Further: Achieving Consistent Performance on Atari
- Deep Reinforcement Learning Control of Quantum Cartpoles
- Towards Characterizing Divergence in Deep Q-Learning
- Convergent Tree Backup and Retrace with Function Approximation
- A Kernel Loss for Solving the Bellman Equation
- LaProp: Separating Momentum and Adaptivity in Adam
- An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay
- Gradient Temporal-Difference Learning with Regularized Corrections
- Breaking the Deadly Triad with a Target Network