Universal Off-Policy Evaluation
arXiv:2104.12820
Abstract
When faced with sequential decision-making problems, it is often useful to be able to predict what would happen if decisions were made using a new policy. Those predictions must often be based on data collected under some previously used decision-making rule. Many previous methods enable such off-policy (or counterfactual) estimation of the expected value of a performance measure called the return. In this paper, we take the first steps towards a universal off-policy estimator (UnO) -- one that provides off-policy estimates and high-confidence bounds for any parameter of the return distribution. We use UnO for estimating and simultaneously bounding the mean, variance, quantiles/median, inter-quantile range, CVaR, and the entire cumulative distribution of returns. Finally, we also discuss Uno's applicability in various settings, including fully observable, partially observable (i.e., with unobserved confounders), Markovian, non-Markovian, stationary, smoothly non-stationary, and discrete distribution shifts.
arXiv commentAccepted at Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS 2021)
References in corpus (25)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Implicit Quantile Networks for Distributional Reinforcement Learning
- A Distributional Perspective on Reinforcement Learning
- Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
- An Analysis of Categorical Distributional Reinforcement Learning
- Parametric Return Density Estimation for Reinforcement Learning
- Deep Reinforcement Learning amidst Lifelong Non-Stationarity
- Reinforcement Learning via Fenchel-Rockafellar Duality
- Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation
- CoinDICE: Off-Policy Confidence Interval Estimation
- Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding
- Policy Evaluation Networks
- Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
- Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation
- Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning
- Off-Policy Risk Assessment in Contextual Bandits
- Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weighting
- Towards Safe Policy Improvement for Non-Stationary MDPs
- Offline Policy Selection under Uncertainty
- Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders
- Optimizing for the Future in Non-Stationary MDPs
- Importance Sampling Policy Evaluation with an Estimated Behavior Policy
- Bayesian Robust Optimization for Imitation Learning
- Reinforcement Learning for Strategic Recommendations
- Parameter-Based Value Functions