A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variants

Abstract

This paper develops an unified framework to study finite-sample convergence guarantees of a large class of value-based asynchronous reinforcement learning (RL) algorithms. We do this by first reformulating the RL algorithms as \textit{Markovian Stochastic Approximation} (SA) algorithms to solve fixed-point equations. We then develop a Lyapunov analysis and derive mean-square error bounds on the convergence of the Markovian SA. Based on this result, we establish finite-sample mean-square convergence bounds for asynchronous RL algorithms such as $Q$-learning, $n$-step TD, TD$(\lambda )$, and off-policy TD algorithms including V-trace. As a by-product, by analyzing the convergence bounds of $n$-step TD and TD$(\lambda )$, we provide theoretical insights into the bias-variance trade-off, i.e., efficiency of bootstrapping in RL. This was first posed as an open problem in (Sutton, 1999).

References in corpus (9)

• An Analysis of Reinforcement Learning with Function Approximation
• A Theoretical Analysis of Deep Q-Learning
• Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learning
• Stochastic approximation with cone-contractive operators: Sharp ${\ell }_{\infty }$-bounds for $Q$-learning
• Q($\lambda$) with Off-Policy Corrections
• Finite-Time Analysis for Double Q-learning
• A Unified Switching System Perspective and O.D.E. Analysis of Q-Learning Algorithms
• Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithm
• Optimal Matrix Momentum Stochastic Approximation and Applications to Q-learning

Cited by in corpus (4)

• Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysis
• Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithm
• Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operators
• Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch