Stein Variational Policy Gradient

Abstract

Policy gradient methods have been successfully applied to many complex reinforcement learning problems. However, policy gradient methods suffer from high variance, slow convergence, and inefficient exploration. In this work, we introduce a maximum entropy policy optimization framework which explicitly encourages parameter exploration, and show that this framework can be reduced to a Bayesian inference problem. We then propose a novel Stein variational policy gradient method (SVPG) which combines existing policy gradient methods and a repulsive functional to generate a set of diverse but well-behaved policies. SVPG is robust to initialization and can easily be implemented in a parallel manner. On continuous control problems, we find that implementing SVPG on top of REINFORCE and advantage actor-critic algorithms improves both average return and data efficiency.

References in corpus (4)

• Benchmarking Deep Reinforcement Learning for Continuous Control
• VIME: Variational Information Maximizing Exploration
• Deep Exploration via Bootstrapped DQN
• Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm

Cited by in corpus (25)

• Bayesian Model-Agnostic Meta-Learning
• Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
• Implicit Generation and Generalization in Energy-Based Models
• A Stein variational Newton method
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• An introduction to variational inference in Geophysical inverse problems
• Advances in Variational Inference
• Stein Variational Gradient Descent Without Gradient
• Learning Self-Imitating Diverse Policies
• Stein Variational Model Predictive Control
• Robust Reinforcement Learning via Adversarial training with Langevin Dynamics
• Distributionally Adversarial Attack
• Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning
• Stein Self-Repulsive Dynamics: Benefits From Past Samples
• Wasserstein variational gradient descent: From semi-discrete optimal transport to ensemble variational inference
• Stochastic Particle-Optimization Sampling and the Non-Asymptotic Convergence Theory
• Understanding MCMC Dynamics as Flows on the Wasserstein Space
• Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization
• Loss function based second-order Jensen inequality and its application to particle variational inference
• Application of variational policy gradient to atomic-scale materials synthesis
• Maximum Entropy Diverse Exploration: Disentangling Maximum Entropy Reinforcement Learning
• Improving width-based planning with compact policies
• Scalable Approximate Inference and Some Applications
• Doubly Robust Stein-Kernelized Monte Carlo Estimator: Simultaneous Bias-Variance Reduction and Supercanonical Convergence
• Variational Inference for Policy Gradient