Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences

Abstract

Approximate Policy Iteration (API) algorithms alternate between (approximate) policy evaluation and (approximate) greedification. Many different approaches have been explored for approximate policy evaluation, but less is understood about approximate greedification and what choices guarantee policy improvement. In this work, we investigate approximate greedification when reducing the KL divergence between the parameterized policy and the Boltzmann distribution over action values. In particular, we investigate the difference between the forward and reverse KL divergences, with varying degrees of entropy regularization. We show that the reverse KL has stronger policy improvement guarantees, but that reducing the forward KL can result in a worse policy. We also demonstrate, however, that a large enough reduction of the forward KL can induce improvement under additional assumptions. Empirically, we show on simple continuous-action environments that the forward KL can induce more exploration, but at the cost of a more suboptimal policy. No significant differences were observed in the discrete-action setting or on a suite of benchmark problems. Throughout, we highlight that many policy gradient methods can be seen as an instance of API, with either the forward or reverse KL for the policy update, and discuss next steps for understanding and improving our policy optimization algorithms.

arXiv commentUpdated the paper with more theory in Section 5 and moved some experiments to the Appendix

References in corpus (25)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Reinforcement Learning with Deep Energy-Based Policies
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Deep Reinforcement Learning that Matters
• Maximum a Posteriori Policy Optimisation
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
• A unified view of entropy-regularized Markov decision processes
• A Theory of Regularized Markov Decision Processes
• On the Global Convergence Rates of Softmax Policy Gradient Methods
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Learning to Generalize from Sparse and Underspecified Rewards
• MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments
• Approximate Policy Iteration Schemes: A Comparison
• VIREL: A Variational Inference Framework for Reinforcement Learning
• A Closer Look at Deep Policy Gradients
• Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies
• The Value Function Polytope in Reinforcement Learning
• Entropy Regularization with Discounted Future State Distribution in Policy Gradient Methods
• Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergence
• Improving Policy Gradient by Exploring Under-appreciated Rewards
• Learning Policies through Quantile Regression

Cited by in corpus (2)

• Shared learning of powertrain control policies for vehicle fleets
• AdvNF: Reducing Mode Collapse in Conditional Normalising Flows using Adversarial Learning