DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections

Abstract

In many real-world reinforcement learning applications, access to the environment is limited to a fixed dataset, instead of direct (online) interaction with the environment. When using this data for either evaluation or training of a new policy, accurate estimates of discounted stationary distribution ratios -- correction terms which quantify the likelihood that the new policy will experience a certain state-action pair normalized by the probability with which the state-action pair appears in the dataset -- can improve accuracy and performance. In this work, we propose an algorithm, DualDICE, for estimating these quantities. In contrast to previous approaches, our algorithm is agnostic to knowledge of the behavior policy (or policies) used to generate the dataset. Furthermore, it eschews any direct use of importance weights, thus avoiding potential optimization instabilities endemic of previous methods. In addition to providing theoretical guarantees, we present an empirical study of our algorithm applied to off-policy policy evaluation and find that our algorithm significantly improves accuracy compared to existing techniques.

arXiv commentAppearing in NeurIPS 2019 Vancouver, Canada

References in corpus (13)

• Playing Atari with Deep Reinforcement Learning
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• Deep Reinforcement Learning for Dialogue Generation
• Doubly Robust Policy Evaluation and Learning
• Learning Dexterous In-Hand Manipulation
• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
• Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Stochastic Variance Reduction Methods for Policy Evaluation
• Consistent On-Line Off-Policy Evaluation
• Learning from Conditional Distributions via Dual Embeddings
• Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift

Cited by in corpus (48)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• MOReL : Model-Based Offline Reinforcement Learning
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• d3rlpy: An Offline Deep Reinforcement Learning Library
• Minimax Weight and Q-Function Learning for Off-Policy Evaluation
• Is Pessimism Provably Efficient for Offline RL?
• Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research
• Model-Based Offline Planning
• CoinDICE: Off-Policy Confidence Interval Estimation
• Improved Algorithms for Convex-Concave Minimax Optimization
• Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy
• Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
• Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
• Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning
• GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values
• Bootstrapping Fitted Q-Evaluation for Off-Policy Inference
• An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Stable Policy Optimization via Off-Policy Divergence Regularization
• Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
• Nearly Horizon-Free Offline Reinforcement Learning
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• Average-Reward Off-Policy Policy Evaluation with Function Approximation
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
• Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning
• On Instrumental Variable Regression for Deep Offline Policy Evaluation
• Truncated Emphatic Temporal Difference Methods for Prediction and Control
• Learning Retrospective Knowledge with Reverse Reinforcement Learning
• SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matching
• On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction
• Reward is enough for convex MDPs
• Supervised Off-Policy Ranking
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• Curriculum Offline Imitation Learning
• UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning
• Sample Complexity of Offline Reinforcement Learning with Deep ReLU Networks
• Pessimistic Model Selection for Offline Deep Reinforcement Learning
• Agnostic Reinforcement Learning with Low-Rank MDPs and Rich Observations
• Active Offline Policy Selection
• Variance-Aware Off-Policy Evaluation with Linear Function Approximation
• Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach
• Conditional Importance Sampling for Off-Policy Learning