Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features

Jiuqi Wang, Shangtong Zhang

Temporal difference (TD) learning with linear function approximation (linear TD) is a classic and powerful prediction algorithm in reinforcement learning. While it is well-understo…