Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation

Abstract

In reinforcement learning, it is typical to use the empirically observed transitions and rewards to estimate the value of a policy via either model-based or Q-fitting approaches. Although straightforward, these techniques in general yield biased estimates of the true value of the policy. In this work, we investigate the potential for statistical bootstrapping to be used as a way to take these biased estimates and produce calibrated confidence intervals for the true value of the policy. We identify conditions - specifically, sufficient data size and sufficient coverage - under which statistical bootstrapping in this setting is guaranteed to yield correct confidence intervals. In practical situations, these conditions often do not hold, and so we discuss and propose mechanisms that can be employed to mitigate their effects. We evaluate our proposed method and show that it can yield accurate confidence intervals in a variety of conditions, including challenging continuous control environments and small data regimes.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• Doubly Robust Policy Evaluation and Learning
• Deep Exploration via Randomized Value Functions
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation
• Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health

Cited by in corpus (8)

• Benchmarks for Deep Off-Policy Evaluation
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• Offline Policy Selection under Uncertainty
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Universal Off-Policy Evaluation
• Debiasing Samples from Online Learning Using Bootstrap
• Supervised Off-Policy Ranking
• Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Bounds