Minimax Weight and Q-Function Learning for Off-Policy Evaluation

Abstract

We provide theoretical investigations into off-policy evaluation in reinforcement learning using function approximators for (marginalized) importance weights and value functions. Our contributions include: (1) A new estimator, MWL, that directly estimates importance ratios over the state-action distributions, removing the reliance on knowledge of the behavior policy as in prior work (Liu et al., 2018). (2) Another new estimator, MQL, obtained by swapping the roles of importance weights and value-functions in MWL. MQL has an intuitive interpretation of minimizing average Bellman errors and can be combined with MWL in a doubly robust manner. (3) Several additional results that offer further insights into these methods, including the sample complexity analyses of MWL and MQL, their asymptotic optimality in the tabular setting, how the learned importance weights depend the choice of the discriminator class, and how our methods provide a unified view of some old and new algorithms in RL.

References in corpus (11)

• Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
• Information-Theoretic Considerations in Batch Reinforcement Learning
• A Kernel Loss for Solving the Bellman Equation
• On Oracle-Efficient PAC RL with Rich Observations
• Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
• On Value Functions and the Agent-Environment Boundary

Cited by in corpus (11)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• What are the Statistical Limits of Offline RL with Linear Function Approximation?
• Off-Policy Evaluation via the Regularized Lagrangian
• Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
• Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy
• Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• Average-Reward Off-Policy Policy Evaluation with Function Approximation
• Deeply-Debiased Off-Policy Interval Estimation
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation