Stochastic Recursive Momentum for Policy Gradient Methods

Abstract

In this paper, we propose a novel algorithm named STOchastic Recursive Momentum for Policy Gradient (STORM-PG), which operates a SARAH-type stochastic recursive variance-reduced policy gradient in an exponential moving average fashion. STORM-PG enjoys a provably sharp $O(1/{\epsilon }^3)$ sample complexity bound for STORM-PG, matching the best-known convergence rate for policy gradient algorithm. In the mean time, STORM-PG avoids the alternations between large batches and small batches which persists in comparable variance-reduced policy gradient methods, allowing considerably simpler parameter tuning. Numerical experiments depicts the superiority of our algorithm over comparative policy gradient algorithms.

References in corpus (10)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Deep Reinforcement Learning that Matters
• SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
• Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
• An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradient
• Stochastic Variance-Reduced Policy Gradient
• Stochastic Variance Reduction for Policy Gradient Estimation

Cited by in corpus (9)

• Efficient Privacy-Preserving Stochastic Nonconvex Optimization
• A general sample complexity analysis of vanilla policy gradient
• Stochastic Recursive Momentum Method for Non-Convex Compositional Optimization
• A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms
• Variance-Reduced Off-Policy Memory-Efficient Policy Search
• ErrorCompensatedX: error compensation for variance reduced algorithms
• Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds
• On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction
• Adaptive Stochastic ADMM for Decentralized Reinforcement Learning in Edge Industrial IoT