Partially Observable Markov Decision Processes (POMDPs) and Robotics

Abstract

Planning under uncertainty is critical to robotics. The Partially Observable Markov Decision Process (POMDP) is a mathematical framework for such planning problems. It is powerful due to its careful quantification of the non-deterministic effects of actions and partial observability of the states. But precisely because of this, POMDP is notorious for its high computational complexity and deemed impractical for robotics. However, since early 2000, POMDPs solving capabilities have advanced tremendously, thanks to sampling-based approximate solvers. Although these solvers do not generate the optimal solution, they can compute good POMDP solutions that significantly improve the robustness of robotics systems within reasonable computational resources, thereby making POMDPs practical for many realistic robotics problems. This paper presents a review of POMDPs, emphasizing computational issues that have hindered its practicality in robotics and ideas in sampling-based solvers that have alleviated such difficulties, together with lessons learned from applying POMDPs to physical robots.

References in corpus (3)

• Value-Function Approximations for Partially Observable Markov Decision Processes
• Bayesian Reinforcement Learning: A Survey
• Speeding Up the Convergence of Value Iteration in Partially Observable Markov Decision Processes

Cited by in corpus (7)

• Searching for a source without gradients: how good is infotaxis and how to beat it
• Deep reinforcement learning for the olfactory search POMDP: a quantitative benchmark
• Observation-Augmented Contextual Multi-Armed Bandits for Robotic Search and Exploration
• Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency
• Learning A Unified Risk Map for Autonomous Driving in Partially Observable Environments
• Active Inference in Contextual Multi-Armed Bandits for Autonomous Robotic Exploration
• Rollout Heuristics for Online Stochastic Contingent Planning