12 papers
Spectral Analysis of Dueling Q-Learning
Donghwan Lee
Q-learning is a fundamental algorithm in reinforcement learning (RL) for solving discounted Markov decision processes (MDPs) when the transition kernel is unknown. The deep Q-netwo…
Heavy-Ball Q-Learning with Residual Weighting Correction
Donghwan Lee
This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL) and establishes convergence of its deterministic mean dynamics. It also identifies cond…
Sign-Separated Asymmetric Finite-Time Error Analysis of Q-Learning
Donghwan Lee
Q-learning is known to suffer from overestimation bias: because the Bellman update maximizes noisy or imperfect action-value estimates, positive errors can be selected and propagat…
A Switching System Theory of Q-Learning with Linear Function Approximation
Donghwan Lee, Han-Dong Lim
Q-learning is a fundamental algorithmic primitive in reinforcement learning. This paper develops a new framework for analyzing linear Q-learning from a switching linear system (SLS…
Switching Theory for Q-Learning
Donghwan Lee
Q-learning is a fundamental algorithmic primitive in reinforcement learning. This paper develops a new framework for analyzing constant step-size tabular Q-learning from a switchin…
Spectral Analysis of Heavy-Ball Q-value Iteration
Donghwan Lee
We study the convergence and acceleration of Qvalue iteration (QVI) with heavy-ball momentum. Although acceleration of value iteration has been studied extensively, there has been…