Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settings

Abstract

Reinforcement learning is a general technique that allows an agent to learn an optimal policy and interact with an environment in sequential decision making problems. The goodness of a policy is measured by its value function starting from some initial state. The focus of this paper is to construct confidence intervals (CIs) for a policy's value in infinite horizon settings where the number of decision points diverges to infinity. We propose to model the action-value state function (Q-function) associated with a policy based on series/sieve method to derive its confidence interval. When the target policy depends on the observed data as well, we propose a SequentiAl Value Evaluation (SAVE) method to recursively update the estimated policy and its value estimator. As long as either the number of trajectories or the number of decision points diverges to infinity, we show that the proposed CI achieves nominal coverage even in cases where the optimal policy is not unique. Simulation studies are conducted to back up our theoretical findings. We apply the proposed method to a dataset from mobile health studies and find that reinforcement learning algorithms could help improve patient's health status. A Python implementation of the proposed procedure is available at https://github.com/shengzhang37/SAVE.

References in corpus (8)

• Basic Properties of Strong Mixing Conditions. A Survey and Some Open Questions
• Fast learning rates for plug-in classifiers
• Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy
• Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising
• Optimal Uniform Convergence Rates and Asymptotic Normality for Series Estimators Under Weak Dependence and Weak Conditions
• A Theoretical Analysis of Deep Q-Learning
• Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation
• Fast Rates for the Regret of Offline Reinforcement Learning

Cited by in corpus (5)

• Deeply-Debiased Off-Policy Interval Estimation
• Projected State-action Balancing Weights for Offline Reinforcement Learning
• Robust Batch Policy Learning in Markov Decision Processes
• Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learning
• Pessimistic Model Selection for Offline Deep Reinforcement Learning