Hierarchical POMDP Controller Optimization by Likelihood Maximization

Abstract

Planning can often be simpli ed by decomposing the task into smaller tasks arranged hierarchically. Charlin et al. [4] recently showed that the hierarchy discovery problem can be framed as a non-convex optimization problem. However, the inherent computational di culty of solving such an optimization problem makes it hard to scale to realworld problems. In another line of research, Toussaint et al. [18] developed a method to solve planning problems by maximumlikelihood estimation. In this paper, we show how the hierarchy discovery problem in partially observable domains can be tackled using a similar maximum likelihood approach. Our technique rst transforms the problem into a dynamic Bayesian network through which a hierarchical structure can naturally be discovered while optimizing the policy. Experimental results demonstrate that this approach scales better than previous techniques based on non-convex optimization.

Appears in Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence (UAI2008)

References in corpus (2)

• Learning Finite-State Controllers for Partially Observable Environments
• Policy-contingent abstraction for robust robot control

Cited by in corpus (8)

• Game-Theoretic Multiagent Reinforcement Learning
• Anytime Planning for Decentralized POMDPs using Expectation Maximization
• New inference strategies for solving Markov Decision Processes using reversible jump MCMC
• Learning Deep Neural Network Policies with Continuous Memory States
• Active Inference or Control as Inference? A Unifying View
• Program Synthesis Guided Reinforcement Learning for Partially Observed Environments
• How memory architecture affects learning in a simple POMDP: the two-hypothesis testing problem
• How Morphological Computation shapes Integrated Information in Embodied Agents