OTD: (Near)-Optimal Off-Policy TD Learning
arXiv:1704.05147
Abstract
Temporal difference learning and Residual Gradient methods are the most widely used temporal difference based learning algorithms; however, it has been shown that none of their objective functions is optimal w.r.t approximating the true value function . Two novel algorithms are proposed to approximate the true value function . This paper makes the following contributions: (1) A batch algorithm that can help find the approximate optimal off-policy prediction of the true value function . (2) A linear computational cost (per step) near-optimal algorithm that can learn from a collection of off-policy samples. (3) A new perspective of the emphatic temporal difference learning which bridges the gap between off-policy optimality and off-policy stability.
arXiv comment10 pages, 7 figures
References in corpus (3)
- Finite-Sample Analysis of Proximal Gradient TD Algorithms
- Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view
- Weak Convergence Properties of Constrained Emphatic Temporal-difference Learning with Constant and Slowly Diminishing Stepsize