Measurable Monte Carlo Search Error Bounds
arXiv:2106.04715
Abstract
Monte Carlo planners can often return sub-optimal actions, even if they are guaranteed to converge in the limit of infinite samples. Known asymptotic regret bounds do not provide any way to measure confidence of a recommended action at the conclusion of search. In this work, we prove bounds on the sub-optimality of Monte Carlo estimates for non-stationary bandits and Markov decision processes. These bounds can be directly computed at the conclusion of the search and do not require knowledge of the true action-value. The presented bound holds for general Monte Carlo solvers meeting mild convergence conditions. We empirically test the tightness of the bounds through experiments on a multi-armed bandit and a discrete Markov decision process for both a simple solver and Monte Carlo tree search.
arXiv comment9 pages, submitted to ALT 2022
References in corpus (6)
- Empirical Bernstein Bounds and Sample Variance Penalization
- Contextual Bandit Algorithms with Supervised Learning Guarantees
- Combining Neural Networks and Tree Search for Task and Motion Planning in Challenging Environments
- Blazing the trails before beating the path: Sample-efficient Monte-Carlo planning
- A central limit theorem for temporally non-homogenous Markov chains with applications to dynamic programming
- Planning in entropy-regularized Markov decision processes and games