An operator view of policy gradient methods

Abstract

We cast policy gradient methods as the repeated application of two operators: a policy improvement operator $\mathcal{I}$, which maps any policy $\pi$ to a better one $\mathcal{I}\pi$, and a projection operator $\mathcal{P}$, which finds the best approximation of $\mathcal{I}\pi$ in the set of realizable policies. We use this framework to introduce operator-based versions of traditional policy gradient methods such as REINFORCE and PPO, which leads to a better understanding of their original counterparts. We also use the understanding we develop of the role of $\mathcal{I}$ and $\mathcal{P}$ to propose a new global lower bound of the expected return. This new perspective allows us to further bridge the gap between policy-based and value-based methods, showing how REINFORCE and the Bellman optimality operator, for example, can be seen as two sides of the same coin.

arXiv commentNeurIPS 2020

References in corpus (8)

• Trust Region Policy Optimization
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Maximum a Posteriori Policy Optimisation
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Imitation Learning as $f$-Divergence Minimization
• Taylor Expansion Policy Optimization

Cited by in corpus (2)

• Learning and Planning in Complex Action Spaces
• Reinforcement Learning for Vision-based Object Manipulation with Non-parametric Policy and Action Primitives