Minimax Value Interval for Off-Policy Evaluation and Policy Optimization

Abstract

We study minimax methods for off-policy evaluation (OPE) using value functions and marginalized importance weights. Despite that they hold promises of overcoming the exponential variance in traditional importance sampling, several key problems remain: (1) They require function approximation and are generally biased. For the sake of trustworthy OPE, is there anyway to quantify the biases? (2) They are split into two styles ("weight-learning" vs "value-learning"). Can we unify them? In this paper we answer both questions positively. By slightly altering the derivation of previous methods (one from each style; Uehara et al., 2020), we unify them into a single value interval that comes with a special type of double robustness: when either the value-function or the importance-weight class is well specified, the interval is valid and its length quantifies the misspecification of the other class. Our interval also provides a unified view of and new insights to some recent methods, and we further explore the implications of our results on exploration and exploitation in off-policy policy optimization with insufficient data coverage.

References in corpus (17)

• Playing Atari with Deep Reinforcement Learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Doubly Robust Policy Evaluation and Learning
• Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• GenDICE: Generalized Offline Estimation of Stationary Values
• Information-Theoretic Considerations in Batch Reinforcement Learning
• Primal-Dual $\pi$ Learning: Sample Complexity and Sublinear Run Time for Ergodic Markov Decision Problems
• Reinforcement Learning via Fenchel-Rockafellar Duality
• A Kernel Loss for Solving the Bellman Equation
• CoinDICE: Off-Policy Confidence Interval Estimation
• Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health
• Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Sampling

Cited by in corpus (11)

• What are the Statistical Limits of Offline RL with Linear Function Approximation?
• Batch Value-function Approximation with Only Realizability
• Minimax Weight and Q-Function Learning for Off-Policy Evaluation
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• CoinDICE: Off-Policy Confidence Interval Estimation
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
• Accountable Off-Policy Evaluation With Kernel Bellman Statistics
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Universal Off-Policy Evaluation
• On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction