Off-Policy Evaluation via the Regularized Lagrangian

Abstract

The recently proposed distribution correction estimation (DICE) family of estimators has advanced the state of the art in off-policy evaluation from behavior-agnostic data. While these estimators all perform some form of stationary distribution correction, they arise from different derivations and objective functions. In this paper, we unify these estimators as regularized Lagrangians of the same linear program. The unification allows us to expand the space of DICE estimators to new alternatives that demonstrate improved performance. More importantly, by analyzing the expanded space of estimators both mathematically and empirically we find that dual solutions offer greater flexibility in navigating the tradeoff between optimization stability and estimation bias, and generally provide superior estimates in practice.

References in corpus (12)

• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
• Finite-Sample Analysis of Proximal Gradient TD Algorithms
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
• Stochastic Variance Reduction Methods for Policy Evaluation
• GenDICE: Generalized Offline Estimation of Stationary Values
• Scalable Bilinear $\pi$ Learning Using State and Action Features
• Minimax Weight and Q-Function Learning for Off-Policy Evaluation
• Randomized Linear Programming Solves the Discounted Markov Decision Problem In Nearly-Linear (Sometimes Sublinear) Running Time
• Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation
• Faster saddle-point optimization for solving large-scale Markov decision processes

Cited by in corpus (17)

• Benchmarks for Deep Off-Policy Evaluation
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
• Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality
• Nearly Horizon-Free Offline Reinforcement Learning
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Infinite-Horizon Offline Reinforcement Learning with Linear Function Approximation: Curse of Dimensionality and Algorithm
• On Instrumental Variable Regression for Deep Offline Policy Evaluation
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• Supervised Off-Policy Ranking
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction
• Reward is enough for convex MDPs
• Active Offline Policy Selection
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach