12 papers
Spectral Analysis of Heavy-Ball Q-value Iteration
Donghwan Lee
We study the convergence and acceleration of Qvalue iteration (QVI) with heavy-ball momentum. Although acceleration of value iteration has been studied extensively, there has been…
Spectral Analysis of Dueling Q-Learning
Donghwan Lee
Q-learning is a fundamental algorithm in reinforcement learning (RL) for solving discounted Markov decision processes (MDPs) when the transition kernel is unknown. The deep Q-netwo…
Heavy-Ball Q-Learning with Residual Weighting Correction
Donghwan Lee
This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL) and establishes convergence of its deterministic mean dynamics. It also identifies cond…
Geometrically Averaged Hard Target Updates for Linear Q-Learning
Donghwan Lee
Periodic hard target updates are among the most common stabilization devices in modern deep Q-learning. Recent studies suggest that target updates can improve stability in Q-learni…
Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamics
Donghwan Lee
Periodic target updates in Q-learning and soft target updates in actor-critic methods are empirically well established stabilization mechanisms, but their precise theoretical expla…
Switching-Geometry Analysis of Deflated Q-Value Iteration
Donghwan Lee
This paper develops a joint spectral radius (JSR) framework for analyzing rank-one deflated Q-value iteration (Q-VI) in discounted Markov decision process control. Focusing on an a…
Soft Deterministic Policy Gradient with Gaussian Smoothing
Hyunjun Na, Donghwan Lee
Deterministic policy gradient (DPG) is widely utilized for continuous control; however, it inherently relies on the differentiability of the critic with respect to the action durin…
Sign-Separated Asymmetric Finite-Time Error Analysis of Q-Learning
Donghwan Lee
Q-learning is known to suffer from overestimation bias: because the Bellman update maximizes noisy or imperfect action-value estimates, positive errors can be selected and propagat…
A Switching System Theory of Q-Learning with Linear Function Approximation
Donghwan Lee, Han-Dong Lim
Q-learning is a fundamental algorithmic primitive in reinforcement learning. This paper develops a new framework for analyzing linear Q-learning from a switching linear system (SLS…
Switching Theory for Q-Learning
Donghwan Lee
Q-learning is a fundamental algorithmic primitive in reinforcement learning. This paper develops a new framework for analyzing constant step-size tabular Q-learning from a switchin…
R-GTD: A Geometric Analysis of Gradient Temporal-Difference Learning in Singular Regimes
Hyunjun Na, Donghwan Lee
Gradient temporal-difference (GTD) learning algorithms are widely used for off-policy policy evaluation with function approximation. However, existing convergence analyses rely on…
Finite-Time Analysis of Simultaneous Double Q-learning
Hyunjun Na, Donghwan Lee
-learning is one of the most fundamental reinforcement learning (RL) algorithms. Despite its widespread success in various applications, it is prone to overestimation bias in th…