Adaptive Sampling for Best Policy Identification in Markov Decision Processes
arXiv:2009.13405
Abstract
We investigate the problem of best-policy identification in discounted Markov Decision Processes (MDPs) when the learner has access to a generative model. The objective is to devise a learning algorithm returning the best policy as early as possible. We first derive a problem-specific lower bound of the sample complexity satisfied by any learning algorithm. This lower bound corresponds to an optimal sample allocation that solves a non-convex program, and hence, is hard to exploit in the design of efficient algorithms. We then provide a simple and tight upper bound of the sample complexity lower bound, whose corresponding nearly-optimal sample allocation becomes explicit. The upper bound depends on specific functionals of the MDP such as the sub-optimality gaps and the variance of the next-state value function, and thus really captures the hardness of the MDP. Finally, we devise KLB-TS (KL Ball Track-and-Stop), an algorithm tracking this nearly-optimal allocation, and provide asymptotic guarantees for its sample complexity (both almost surely and in expectation). The advantages of KLB-TS against state-of-the-art algorithms are discussed and illustrated numerically.
arXiv comment43 pages
References in corpus (6)
- On the Complexity of Best Arm Identification in Multi-Armed Bandit Models
- Optimal Best Arm Identification with Fixed Confidence
- Model-Based Reinforcement Learning with a Generative Model is Minimax Optimal
- Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model
- Mixture Martingales Revisited with Applications to Sequential Tests and Confidence Intervals
- Planning in Markov Decision Processes with Gap-Dependent Sample Complexity