Geometry and Determinism of Optimal Stationary Control in Partially Observable Markov Decision Processes

Abstract

It is well known that for any finite state Markov decision process (MDP) there is a memoryless deterministic policy that maximizes the expected reward. For partially observable Markov decision processes (POMDPs), optimal memoryless policies are generally stochastic. We study the expected reward optimization problem over the set of memoryless stochastic policies. We formulate this as a constrained linear optimization problem and develop a corresponding geometric framework. We show that any POMDP has an optimal memoryless policy of limited stochasticity, which allows us to reduce the dimensionality of the search space. Experiments demonstrate that this approach enables better and faster convergence of the policy gradient on the evaluated systems.

25 pages, 7 figures

Cited by in corpus (5)

• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• Policy Gradient in Partially Observable Environments: Approximation and Convergence
• Open Problem: Approximate Planning of POMDPs in the class of Memoryless Policies
• The Geometry of Memoryless Stochastic Policy Optimization in Infinite-Horizon POMDPs
• Geometry of Policy Improvement