A Hybrid Stochastic Policy Gradient Algorithm for Reinforcement Learning

Abstract

We propose a novel hybrid stochastic policy gradient estimator by combining an unbiased policy gradient estimator, the REINFORCE estimator, with another biased one, an adapted SARAH estimator for policy optimization. The hybrid policy gradient estimator is shown to be biased, but has variance reduced property. Using this estimator, we develop a new Proximal Hybrid Stochastic Policy Gradient Algorithm (ProxHSPGA) to solve a composite policy optimization problem that allows us to handle constraints or regularizers on the policy parameters. We first propose a single-looped algorithm then introduce a more practical restarting variant. We prove that both algorithms can achieve the best-known trajectory complexity $\mathcal{O}\left({\epsilon }^{-3}\right)$ to attain a first-order stationary point for the composite problem which is better than existing REINFORCE/GPOMDP $\mathcal{O}\left({\epsilon }^{-4}\right)$ and SVRPG $\mathcal{O}\left({\epsilon }^{-10/3}\right)$ in the non-composite setting. We evaluate the performance of our algorithm on several well-known examples in reinforcement learning. Numerical results show that our algorithm outperforms two existing methods on these examples. Moreover, the composite settings indeed have some advantages compared to the non-composite ones on certain problems.

arXiv commentAccepted for publication at the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS 2020)

References in corpus (14)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Dueling Network Architectures for Deep Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
• A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization
• Stochastic Recursive Gradient Algorithm for Nonconvex Optimization
• Stochastic Nested Variance Reduction for Nonconvex Optimization
• Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
• Regularization Matters in Policy Optimization
• A Hybrid Stochastic Optimization Framework for Stochastic Composite Nonconvex Optimization

Cited by in corpus (3)

• On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Method
• A general sample complexity analysis of vanilla policy gradient
• Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds