Learning and Planning in Average-Reward Markov Decision Processes

Abstract

We introduce learning and planning algorithms for average-reward MDPs, including 1) the first general proven-convergent off-policy model-free control algorithm without reference states, 2) the first proven-convergent off-policy model-free prediction algorithm, and 3) the first off-policy learning algorithm that converges to the actual value function rather than to the value function plus an offset. All of our algorithms are based on using the temporal-difference error rather than the conventional error when updating the estimate of the average reward. Our proof techniques are a slight generalization of those by Abounadi, Bertsekas, and Borkar (2001). In experiments with an Access-Control Queuing Task, we show some of the difficulties that can arise when using methods that rely on reference states and argue that our new algorithms can be significantly easier to use.

arXiv commentIn Proceedings of ICML 2021

References in corpus (5)

• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Discounted Reinforcement Learning Is Not an Optimization Problem
• Exploration-Enhanced POLITEX
• Average-reward model-free reinforcement learning: a systematic review and literature mapping
• Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation

Cited by in corpus (4)

• Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Tasks
• Average-reward model-free reinforcement learning: a systematic review and literature mapping
• Average-Reward Off-Policy Policy Evaluation with Function Approximation
• Breaking the Deadly Triad with a Target Network