Towards Characterizing Divergence in Deep Q-Learning

Abstract

Deep Q-Learning (DQL), a family of temporal difference algorithms for control, employs three techniques collectively known as the `deadly triad' in reinforcement learning: bootstrapping, off-policy learning, and function approximation. Prior work has demonstrated that together these can lead to divergence in Q-learning algorithms, but the conditions under which divergence occurs are not well-understood. In this note, we give a simple analysis based on a linear approximation to the Q-value updates, which we believe provides insight into divergence under the deadly triad. The central point in our analysis is to consider when the leading order approximation to the deep-Q update is or is not a contraction in the sup norm. Based on this analysis, we develop an algorithm which permits stable deep Q-learning for continuous control without any of the tricks conventionally used (such as target networks, adaptive gradient optimizers, or using multiple Q functions). We demonstrate that our algorithm performs above or near state-of-the-art on standard MuJoCo benchmarks from the OpenAI Gym.

References in corpus (10)

• Continuous control with deep reinforcement learning
• Addressing Function Approximation Error in Actor-Critic Methods
• Dueling Network Architectures for Deep Reinforcement Learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Distributed Prioritized Experience Replay
• Equivalence Between Policy Gradients and Soft Q-Learning
• Observe and Look Further: Achieving Consistent Performance on Atari
• Diagnosing Bottlenecks in Deep Q-learning Algorithms
• Natural Gradient Deep Q-learning

Cited by in corpus (35)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Conservative Q-Learning for Offline Reinforcement Learning
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Real-world Video Adaptation with Reinforcement Learning
• A Fine-Grained Spectral Perspective on Neural Networks
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• The problem with DDPG: understanding failures in deterministic environments with sparse rewards
• Deep Q-learning: a robust control approach
• AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima
• On the Estimation Bias in Double Q-Learning
• Interference and Generalization in Temporal Difference Learning
• Training Larger Networks for Deep Reinforcement Learning
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Optimizing Mixed Autonomy Traffic Flow With Decentralized Autonomous Vehicles and Multi-Agent RL
• Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networks
• Geometric Insights into the Convergence of Nonlinear TD Learning
• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
• Convergent and Efficient Deep Q Network Algorithm
• Breaking the Deadly Triad with a Target Network
• GRAC: Self-Guided and Self-Regularized Actor-Critic
• Qgraph-bounded Q-learning: Stabilizing Model-Free Off-Policy Deep Reinforcement Learning
• Regularized Behavior Value Estimation
• Temporal-difference learning with nonlinear function approximation: lazy training and mean field regimes
• PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning
• Approximating two value functions instead of one: towards characterizing a new family of Deep Reinforcement Learning algorithms
• A Deep Reinforcement Learning Architecture for Multi-stage Optimal Control
• Towards a practical measure of interference for reinforcement learning
• Predictor-Corrector(PC) Temporal Difference(TD) Learning (PCTD)
• Correcting Momentum in Temporal Difference Learning
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• Learning Off-Policy with Online Planning
• Inverse Policy Evaluation for Value-based Sequential Decision-making
• Evolution of Q Values for Deep Q Learning in Stable Baselines