Diagnosing Bottlenecks in Deep Q-learning Algorithms

Abstract

Q-learning methods represent a commonly used class of algorithms in reinforcement learning: they are generally efficient and simple, and can be combined readily with function approximators for deep reinforcement learning (RL). However, the behavior of Q-learning methods with function approximation is poorly understood, both theoretically and empirically. In this work, we aim to experimentally investigate potential issues in Q-learning, by means of a "unit testing" framework where we can utilize oracles to disentangle sources of error. Specifically, we investigate questions related to function approximation, sampling error and nonstationarity, and where available, verify if trends found in oracle settings hold true with modern deep RL methods. We find that large neural network architectures have many benefits with regards to learning stability; offer several practical compensations for overfitting; and develop a novel sampling method based on explicitly compensating for function approximation error that yields fair improvement on high-dimensional continuous control domains.

Cited by in corpus (26)

• Behavior Regularized Offline Reinforcement Learning
• Towards Characterizing Divergence in Deep Q-Learning
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Reward-Conditioned Policies
• On the Estimation Bias in Double Q-Learning
• Training Larger Networks for Deep Reinforcement Learning
• An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay
• Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization
• C-Learning: Learning to Achieve Goals via Recursive Classification
• ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrations
• Reverb: A Framework For Experience Replay
• Qgraph-bounded Q-learning: Stabilizing Model-Free Off-Policy Deep Reinforcement Learning
• Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization
• You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
• On the Expressivity of Neural Networks for Deep Reinforcement Learning
• A New Approach for Tactical Decision Making in Lane Changing: Sample Efficient Deep Q Learning with a Safety Feedback Reward
• Dual Behavior Regularized Reinforcement Learning
• Learning Off-Policy with Online Planning
• Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularization
• Evolution of Q Values for Deep Q Learning in Stable Baselines
• Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning
• Cautious Actor-Critic
• Geometric Value Iteration: Dynamic Error-Aware KL Regularization for Reinforcement Learning
• DROMO: Distributionally Robust Offline Model-based Policy Optimization
• Analysis and Optimisation of Bellman Residual Errors with Neural Function Approximation