The Value Function Polytope in Reinforcement Learning

Abstract

We establish geometric and topological properties of the space of value functions in finite state-action Markov decision processes. Our main contribution is the characterization of the nature of its shape: a general polytope (Aigner et al., 2010). To demonstrate this result, we exhibit several properties of the structural relationship between policies and value functions including the line theorem, which shows that the value functions of policies constrained on all but one state describe a line segment. Finally, we use this novel perspective to introduce visualizations to enhance the understanding of the dynamics of reinforcement learning algorithms.

Cited by in corpus (9)

• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL
• The Value Equivalence Principle for Model-Based Reinforcement Learning
• Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
• Bounding Fixed Points of Set-Based Bellman Operator and Nash Equilibria of Stochastic Games
• GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learning
• The Geometry of Memoryless Stochastic Policy Optimization in Infinite-Horizon POMDPs
• The Information Geometry of Unsupervised Reinforcement Learning
• An Entropy Regularization Free Mechanism for Policy-based Reinforcement Learning