Zap Q-Learning With Nonlinear Function Approximation

Abstract

Zap Q-learning is a recent class of reinforcement learning algorithms, motivated primarily as a means to accelerate convergence. Stability theory has been absent outside of two restrictive classes: the tabular setting, and optimal stopping. This paper introduces a new framework for analysis of a more general class of recursive algorithms known as stochastic approximation. Based on this general theory, it is shown that Zap Q-learning is consistent under a non-degeneracy assumption, even when the function approximation architecture is nonlinear. Zap Q-learning with neural network function approximation emerges as a special case, and is tested on examples from OpenAI Gym. Based on multiple experiments with a range of neural network sizes, it is found that the new algorithms converge quickly and are robust to choice of function approximation architecture.

References in corpus (7)

• Provably Efficient Reinforcement Learning with Linear Function Approximation
• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
• Convergence rate of linear two-time-scale stochastic approximation
• Finite-Sample Analysis of Proximal Gradient TD Algorithms
• Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learning
• Fastest Convergence for Q-learning
• Q-learning with Uniformly Bounded Variance: Large Discounting is Not a Barrier to Fast Learning

Cited by in corpus (4)

• Explicit Mean-Square Error Bounds for Monte-Carlo and Linear Stochastic Approximation
• Breaking the Deadly Triad with a Target Network
• Convex Q-Learning, Part 1: Deterministic Optimal Control
• TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?