Off-Policy Evaluation via the Regularized Lagrangian

Abstract

The recently proposed distribution correction estimation (DICE) family of estimators has advanced the state of the art in off-policy evaluation from behavior-agnostic data. While these estimators all perform some form of stationary distribution correction, they arise from different derivations and objective functions. In this paper, we unify these estimators as regularized Lagrangians of the same linear program. The unification allows us to expand the space of DICE estimators to new alternatives that demonstrate improved performance. More importantly, by analyzing the expanded space of estimators both mathematically and empirically we find that dual solutions offer greater flexibility in navigating the tradeoff between optimization stability and estimation bias, and generally provide superior estimates in practice.

References in corpus (4)

• Finite-Sample Analysis of Proximal Gradient TD Algorithms
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• GenDICE: Generalized Offline Estimation of Stationary Values
• Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation

Cited by in corpus (6)

• Benchmarks for Deep Off-Policy Evaluation
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Infinite-Horizon Offline Reinforcement Learning with Linear Function Approximation: Curse of Dimensionality and Algorithm
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation