New inference strategies for solving Markov Decision Processes using reversible jump MCMC

Abstract

In this paper we build on previous work which uses inferences techniques, in particular Markov Chain Monte Carlo (MCMC) methods, to solve parameterized control problems. We propose a number of modifications in order to make this approach more practical in general, higher-dimensional spaces. We first introduce a new target distribution which is able to incorporate more reward information from sampled trajectories. We also show how to break strong correlations between the policy parameters and sampled trajectories in order to sample more freely. Finally, we show how to incorporate these techniques in a principled manner to obtain estimates of the optimal policy.

Appears in Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI2009)

References in corpus (2)

• PEGASUS: A Policy Search Method for Large MDPs and POMDPs
• Hierarchical POMDP Controller Optimization by Likelihood Maximization

Cited by in corpus (8)

• Portfolio Allocation for Bayesian Optimization
• Bayesian Multi-Scale Optimistic Optimization
• An Entropy Search Portfolio for Bayesian Optimization
• Sequential Monte Carlo Methods for System Identification
• Anytime Planning for Decentralized POMDPs using Expectation Maximization
• Black-Box Policy Search with Probabilistic Programs
• Bayesian Optimization in a Billion Dimensions via Random Embeddings
• Optimal Control of Complex Systems through Variational Inference with a Discrete Event Decision Process