Breaking the Deadly Triad with a Target Network
arXiv:2101.08862
Abstract
The deadly triad refers to the instability of a reinforcement learning algorithm when it employs off-policy learning, function approximation, and bootstrapping simultaneously. In this paper, we investigate the target network as a tool for breaking the deadly triad, providing theoretical support for the conventional wisdom that a target network stabilizes training. We first propose and analyze a novel target network update rule which augments the commonly used Polyak-averaging style update with two projections. We then apply the target network and ridge regularization in several divergent algorithms and show their convergence to regularized TD fixed points. Those algorithms are off-policy with linear function approximation and bootstrapping, spanning both policy evaluation and control, as well as both discounted and average-reward settings. In particular, we provide the first convergent linear -learning algorithms under nonrestrictive and changing behavior policies without bi-level optimization.
arXiv commentICML 2021
References in corpus (23)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Addressing Function Approximation Error in Actor-Critic Methods
- Challenges of Real-World Reinforcement Learning
- Provably Efficient Reinforcement Learning with Linear Function Approximation
- An Analysis of Reinforcement Learning with Function Approximation
- A Theoretical Analysis of Deep Q-Learning
- Deep Reinforcement Learning and the Deadly Triad
- Stochastic Variance Reduction Methods for Policy Evaluation
- Finite-Sample Analysis for SARSA with Linear Function Approximation
- Towards Characterizing Divergence in Deep Q-Learning
- Optimism in Reinforcement Learning with Generalized Linear Function Approximation
- A Finite Time Analysis of Two Time-Scale Actor Critic Methods
- Provably Efficient -learning with Function Approximation via Distribution Shift Error Checking Oracle
- Sample-Optimal Parametric Q-Learning Using Linearly Additive Features
- Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learning
- On Convergence of some Gradient-based Temporal-Differences Algorithms for Off-Policy Learning
- Regularized Off-Policy TD-Learning
- Learning and Planning in Average-Reward Markov Decision Processes
- Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima
- Finite-sample Analysis of Greedy-GQ with Linear Function Approximation under Markovian Noise
- Zap Q-Learning With Nonlinear Function Approximation
- Average-Reward Off-Policy Policy Evaluation with Function Approximation