Stable Policy Optimization via Off-Policy Divergence Regularization

Abstract

Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are among the most successful policy gradient approaches in deep reinforcement learning (RL). While these methods achieve state-of-the-art performance across a wide range of challenging tasks, there is room for improvement in the stabilization of the policy learning and how the off-policy data are used. In this paper we revisit the theoretical foundations of these algorithms and propose a new algorithm which stabilizes the policy improvement through a proximity term that constrains the discounted state-action visitation distribution induced by consecutive policies to be close to one another. This proximity term, expressed in terms of the divergence between the visitation distributions, is learned in an off-policy and adversarial manner. We empirically show that our proposed method can have a beneficial effect on stability and improve final performance in benchmark high-dimensional control tasks.

References in corpus (12)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Reinforcement Learning with Deep Energy-Based Policies
• On Convergence and Stability of GANs
• Maximum a Posteriori Policy Optimisation
• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• A unified view of entropy-regularized Markov decision processes
• On integral probability metrics, ϕ-divergences and binary classification
• A Theory of Regularized Markov Decision Processes
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Divergence-Augmented Policy Optimization

Cited by in corpus (4)

• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
• Qgraph-bounded Q-learning: Stabilizing Model-Free Off-Policy Deep Reinforcement Learning
• Divergence-Regularized Multi-Agent Actor-Critic