Statistically Efficient Off-Policy Policy Gradients
arXiv:2002.04014
Abstract
Policy gradient methods in reinforcement learning update policy parameters by taking steps in the direction of an estimated gradient of policy value. In this paper, we consider the statistically efficient estimation of policy gradients from off-policy data, where the estimation is particularly non-trivial. We derive the asymptotic lower bound on the feasible mean-squared error in both Markov and non-Markov decision processes and show that existing estimators fail to achieve it in general settings. We propose a meta-algorithm that achieves the lower bound without any parametric assumptions and exhibits a unique 3-way double robustness property. We discuss how to estimate nuisances that the algorithm relies on. Finally, we establish guarantees on the rate at which we approach a stationary point when we take steps in the direction of our new estimated policy gradient.
References in corpus (13)
- Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes
- Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
- AlgaeDICE: Policy Gradient from Arbitrary Experience
- Balanced Policy Evaluation and Learning
- Information-Theoretic Considerations in Batch Reinforcement Learning
- Off-Policy Policy Gradient with State Distribution Correction
- Convergence guarantees for a class of non-convex and non-smooth optimization problems
- Confounding-Robust Policy Improvement
- Stochastic Variance-Reduced Policy Gradient
- Learning When-to-Treat Policies
- Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Sampling
- From Importance Sampling to Doubly Robust Policy Gradient
- Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods