A Study of Policy Gradient on a Class of Exactly Solvable Models

Abstract

Policy gradient methods are extensively used in reinforcement learning as a way to optimize expected return. In this paper, we explore the evolution of the policy parameters, for a special class of exactly solvable POMDPs, as a continuous-state Markov chain, whose transition probabilities are determined by the gradient of the distribution of the policy's value. Our approach relies heavily on random walk theory, specifically on affine Weyl groups. We construct a class of novel partially observable environments with controllable exploration difficulty, in which the value distribution, and hence the policy parameter evolution, can be derived analytically. Using these environments, we analyze the probabilistic convergence of policy gradient to different local maxima of the value function. To our knowledge, this is the first approach developed to analytically compute the landscape of policy gradient in POMDPs for a class of such environments, leading to interesting insights into the difficulty of this problem.

References in corpus (14)

• Continuous control with deep reinforcement learning
• On the Convergence of Adam and Beyond
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Distributed Distributional Deterministic Policy Gradients
• Implicit Quantile Networks for Distributional Reinforcement Learning
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
• An Analysis of Categorical Distributional Reinforcement Learning
• Parametric Return Density Estimation for Reinforcement Learning
• Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
• Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies
• Behaviour Suite for Reinforcement Learning
• A Comparative Analysis of Expected and Distributional Reinforcement Learning
• A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithms