GenDICE: Generalized Offline Estimation of Stationary Values
arXiv:2002.09072
Abstract
An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities defined by the stationary distribution of a Markov chain. In many real-world applications, access to the underlying transition operator is limited to a fixed set of data that has already been collected, without additional interaction with the environment being available. We show that consistent estimation remains possible in this challenging scenario, and that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions, derived from fundamental properties of the stationary distribution, and exploiting constraint reformulations based on variational divergence minimization. The resulting algorithm, GenDICE, is straightforward and effective. We prove its consistency under general conditions, provide an error analysis, and demonstrate strong empirical performance on benchmark problems, including off-line PageRank and off-policy policy evaluation.
ICLR 2020
References in corpus (5)
- Doubly Robust Policy Evaluation and Learning
- Learning from Logged Implicit Exploration Data
- On integral probability metrics, ϕ-divergences and binary classification
- Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning
- Learning from Conditional Distributions via Dual Embeddings
Cited by in corpus (7)
- AlgaeDICE: Policy Gradient from Arbitrary Experience
- Reinforcement Learning via Fenchel-Rockafellar Duality
- Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
- Off-Policy Evaluation via the Regularized Lagrangian
- Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders
- Accountable Off-Policy Evaluation With Kernel Bellman Statistics
- Off-Policy Interval Estimation with Lipschitz Value Iteration