GenDICE: Generalized Offline Estimation of Stationary Values

Abstract

An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities defined by the stationary distribution of a Markov chain. In many real-world applications, access to the underlying transition operator is limited to a fixed set of data that has already been collected, without additional interaction with the environment being available. We show that consistent estimation remains possible in this challenging scenario, and that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions, derived from fundamental properties of the stationary distribution, and exploiting constraint reformulations based on variational divergence minimization. The resulting algorithm, GenDICE, is straightforward and effective. We prove its consistency under general conditions, provide an error analysis, and demonstrate strong empirical performance on benchmark problems, including off-line PageRank and off-policy policy evaluation.

arXiv commentICLR 2020

References in corpus (11)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• MCMC using Hamiltonian dynamics
• Doubly Robust Policy Evaluation and Learning
• On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems
• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
• Learning from Logged Implicit Exploration Data
• On integral probability metrics, ϕ-divergences and binary classification
• Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning
• Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation
• Learning from Conditional Distributions via Dual Embeddings

Cited by in corpus (29)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• Off-Policy Evaluation via the Regularized Lagrangian
• CoinDICE: Off-Policy Confidence Interval Estimation
• Batch Policy Learning in Average Reward Markov Decision Processes
• Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
• GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values
• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality
• Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Accountable Off-Policy Evaluation With Kernel Bellman Statistics
• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
• Average-Reward Off-Policy Policy Evaluation with Function Approximation
• Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
• Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning
• Learning Retrospective Knowledge with Reverse Reinforcement Learning
• Regularized Inverse Reinforcement Learning
• PER-ETD: A Polynomially Efficient Emphatic Temporal Difference Learning Method
• Sample Complexity of Offline Reinforcement Learning with Deep ReLU Networks
• On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction
• Supervised Off-Policy Ranking
• Curriculum Offline Imitation Learning
• Variance-Aware Off-Policy Evaluation with Linear Function Approximation
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach
• Off-Policy Interval Estimation with Lipschitz Value Iteration