Robust Batch Policy Learning in Markov Decision Processes
arXiv:2011.04185
Abstract
We study the offline data-driven sequential decision making problem in the framework of Markov decision process (MDP). In order to enhance the generalizability and adaptivity of the learned policy, we propose to evaluate each policy by a set of the average rewards with respect to distributions centered at the policy induced stationary distribution. Given a pre-collected dataset of multiple trajectories generated by some behavior policy, our goal is to learn a robust policy in a pre-specified policy class that can maximize the smallest value of this set. Leveraging the theory of semi-parametric statistics, we develop a statistically efficient policy learning method for estimating the de ned robust optimal policy. A rate-optimal regret bound up to a logarithmic factor is established in terms of total decision points in the dataset.
References in corpus (12)
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Conservative Q-Learning for Offline Reinforcement Learning
- Convergence rate of linear two-time-scale stochastic approximation
- Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach
- Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
- Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes
- Balanced Policy Evaluation and Learning
- Distributionally Robust Reinforcement Learning
- Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settings
- Distributional Robustness and Regularization in Reinforcement Learning
- Batch Policy Learning in Average Reward Markov Decision Processes
- Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health