Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning

Abstract

Off-policy estimation for long-horizon problems is important in many real-life applications such as healthcare and robotics, where high-fidelity simulators may not be available and on-policy evaluation is expensive or impossible. Recently, \cite{liu18breaking} proposed an approach that avoids the \emph{curse of horizon} suffered by typical importance-sampling-based methods. While showing promising results, this approach is limited in practice as it requires data be drawn from the \emph{stationary distribution} of a \emph{known} behavior policy. In this work, we propose a novel approach that eliminates such limitations. In particular, we formulate the problem as solving for the fixed point of a certain operator. Using tools from Reproducing Kernel Hilbert Spaces (RKHSs), we develop a new estimator that computes importance ratios of stationary distributions, without knowledge of how the off-policy data are collected. We analyze its asymptotic consistency and finite-sample generalization. Experiments on benchmarks verify the effectiveness of our approach.

arXiv commentPublished at ICLR 2020

References in corpus (8)

• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Doubly Robust Policy Evaluation and Learning
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
• RecSim: A Configurable Simulation Platform for Recommender Systems
• Primal-Dual $\pi$ Learning: Sample Complexity and Sublinear Run Time for Ergodic Markov Decision Problems
• Stein Variational Gradient Descent as Moment Matching
• Policy Optimization via Importance Sampling

Cited by in corpus (8)

• GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values
• Offline Reinforcement Learning with Soft Behavior Regularization
• Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders
• Accountable Off-Policy Evaluation With Kernel Bellman Statistics
• Average-Reward Off-Policy Policy Evaluation with Function Approximation
• Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Bounds
• Off-Policy Interval Estimation with Lipschitz Value Iteration
• Optimal Mixture Weights for Off-Policy Evaluation with Multiple Behavior Policies