Batch Policy Learning in Average Reward Markov Decision Processes

Abstract

We consider the batch (off-line) policy learning problem in the infinite horizon Markov Decision Process. Motivated by mobile health applications, we focus on learning a policy that maximizes the long-term average reward. We propose a doubly robust estimator for the average reward and show that it achieves semiparametric efficiency. Further we develop an optimization algorithm to compute the optimal policy in a parameterized stochastic policy class. The performance of the estimated policy is measured by the difference between the optimal average reward in the policy class and the average reward of the estimated policy and we establish a finite-sample regret guarantee. The performance of the method is illustrated by simulation studies and an analysis of a mobile health study promoting physical activity.

References in corpus (5)

• Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes
• Efficient augmentation and relaxation learning for individualized treatment rules using observational data
• GenDICE: Generalized Offline Estimation of Stationary Values
• A Batch, Off-Policy, Actor-Critic Algorithm for Optimizing the Average Reward
• Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health

Cited by in corpus (5)

• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Deeply-Debiased Off-Policy Interval Estimation
• Robust Batch Policy Learning in Markov Decision Processes
• Pessimistic Model Selection for Offline Deep Reinforcement Learning