Emphatic Temporal-Difference Learning

Abstract

Emphatic algorithms are temporal-difference learning algorithms that change their effective state distribution by selectively emphasizing and de-emphasizing their updates on different time steps. Recent works by Sutton, Mahmood and White (2015), and Yu (2015) show that by varying the emphasis in a particular way, these algorithms become stable and convergent under off-policy training with linear function approximation. This paper serves as a unified summary of the available results from both works. In addition, we demonstrate the empirical benefits from the flexibility of emphatic algorithms, including state-dependent discounting, state-dependent bootstrapping, and the user-specified allocation of function approximation resources.

9 pages, accepted for presentation at European Workshop on Reinforcement Learning

References in corpus (1)

• An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning

Cited by in corpus (13)

• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Safe and Efficient Off-Policy Reinforcement Learning
• Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning
• On Convergence of Emphatic Temporal-Difference Learning
• Weak Convergence Properties of Constrained Emphatic Temporal-difference Learning with Constant and Slowly Diminishing Stepsize
• Unifying task specification in reinforcement learning
• Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis
• Is the Policy Gradient a Gradient?
• Forethought and Hindsight in Credit Assignment
• PER-ETD: A Polynomially Efficient Emphatic Temporal Difference Learning Method
• Recurrent Value Functions
• Should All Temporal Difference Learning Use Emphasis?
• Emphatic TD Bellman Operator is a Contraction