Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation

Abstract

Infinite horizon off-policy policy evaluation is a highly challenging task due to the excessively large variance of typical importance sampling (IS) estimators. Recently, Liu et al. (2018a) proposed an approach that significantly reduces the variance of infinite-horizon off-policy evaluation by estimating the stationary density ratio, but at the cost of introducing potentially high biases due to the error in density ratio estimation. In this paper, we develop a bias-reduced augmentation of their method, which can take advantage of a learned value function to obtain higher accuracy. Our method is doubly robust in that the bias vanishes when either the density ratio or the value function estimation is perfect. In general, when either of them is accurate, the bias can also be reduced. Both theoretical and empirical results show that our method yields significant advantages over previous methods.

References in corpus (7)

• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• Doubly Robust Policy Evaluation and Learning
• Finite-Sample Analysis of Proximal Gradient TD Algorithms
• Off-Policy Policy Gradient with State Distribution Correction
• Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning
• Consistent On-Line Off-Policy Evaluation
• A Kernel Loss for Solving the Bellman Equation

Cited by in corpus (11)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Is Pessimism Provably Efficient for Offline RL?
• CoinDICE: Off-Policy Confidence Interval Estimation
• Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settings
• Learning and Planning in Average-Reward Markov Decision Processes
• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality
• Average-Reward Off-Policy Policy Evaluation with Function Approximation
• Pessimistic Model Selection for Offline Deep Reinforcement Learning
• A Unified Off-Policy Evaluation Approach for General Value Function
• Stateful Offline Contextual Policy Evaluation and Learning
• Variance-Aware Off-Policy Evaluation with Linear Function Approximation