Average-Reward Off-Policy Policy Evaluation with Function Approximation
arXiv:2101.02808
Abstract
We consider off-policy policy evaluation with function approximation (FA) in average-reward MDPs, where the goal is to estimate both the reward rate and the differential value function. For this problem, bootstrapping is necessary and, along with off-policy learning and FA, results in the deadly triad (Sutton & Barto, 2018). To address the deadly triad, we propose two novel algorithms, reproducing the celebrated success of Gradient TD algorithms in the average-reward setting. In terms of estimating the differential value function, the algorithms are the first convergent off-policy linear function approximation algorithms. In terms of estimating the reward rate, the algorithms are the first convergent off-policy linear function approximation algorithms that do not require estimating the density ratio. We demonstrate empirically the advantage of the proposed algorithms, as well as their nonlinear variants, over a competitive density-ratio-based approach, in a simple domain as well as challenging robot simulation tasks.
ICML 2021
References in corpus (13)
- Challenges of Real-World Reinforcement Learning
- MOPO: Model-based Offline Policy Optimization
- MOReL : Model-Based Offline Reinforcement Learning
- Finite-Sample Analysis of Proximal Gradient TD Algorithms
- AlgaeDICE: Policy Gradient from Arbitrary Experience
- GenDICE: Generalized Offline Estimation of Stationary Values
- Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces
- Off-Policy Policy Gradient with State Distribution Correction
- Consistent On-Line Off-Policy Evaluation
- Minimax Weight and Q-Function Learning for Off-Policy Evaluation
- Learning and Planning in Average-Reward Markov Decision Processes
- Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation
- Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning