Bregman Gradient Policy Optimization
arXiv:2106.12112
Abstract
In the paper, we design a novel Bregman gradient policy optimization framework for reinforcement learning based on Bregman divergences and momentum techniques. Specifically, we propose a Bregman gradient policy optimization (BGPO) algorithm based on the basic momentum technique and mirror descent iteration. Meanwhile, we further propose an accelerated Bregman gradient policy optimization (VR-BGPO) algorithm based on the variance reduced technique. Moreover, we provide a convergence analysis framework for our Bregman gradient policy optimization under the nonconvex setting. We prove that our BGPO achieves a sample complexity of for finding -stationary policy only requiring one trajectory at each iteration, and our VR-BGPO reaches the best known sample complexity of , which also only requires one trajectory at each iteration. In particular, by using different Bregman divergences, our BGPO framework unifies many existing policy optimization algorithms such as the existing (variance reduced) policy gradient algorithms such as natural policy gradient algorithm. Extensive experimental results on multiple reinforcement learning tasks demonstrate the efficiency of our new algorithms.
arXiv commentPublished in ICLR 2022
References in corpus (22)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Trust Region Policy Optimization
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Deep Reinforcement Learning: An Overview
- Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
- SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
- On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
- A unified view of entropy-regularized Markov decision processes
- A Theory of Regularized Markov Decision Processes
- Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy
- Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization
- Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
- Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies
- On the Convergence Rate of Stochastic Mirror Descent for Nonsmooth Nonconvex Optimization
- An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods
- Mirror Descent Policy Optimization
- Efficiently Solving MDPs with Stochastic Mirror Descent
- Divergence-Augmented Policy Optimization
- Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence
- SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients
- Momentum-Based Policy Gradient Methods