Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy

Abstract

Evolution strategy (ES) has been shown great promise in many challenging reinforcement learning (RL) tasks, rivaling other state-of-the-art deep RL methods. Yet, there are two limitations in the current ES practice that may hinder its otherwise further capabilities. First, most current methods rely on Monte Carlo type gradient estimators to suggest search direction, where the policy parameter is, in general, randomly sampled. Due to the low accuracy of such estimators, the RL training may suffer from slow convergence and require more iterations to reach optimal solution. Secondly, the landscape of reward functions can be deceptive and contains many local maxima, causing ES algorithms to prematurely converge and be unable to explore other parts of the parameter space with potentially greater rewards. In this work, we employ a Directional Gaussian Smoothing Evolutionary Strategy (DGS-ES) to accelerate RL training, which is well-suited to address these two challenges with its ability to i) provide gradient estimates with high accuracy, and ii) find nonlocal search direction which lays stress on large-scale variation of the reward function and disregards local fluctuation. Through several benchmark RL tasks demonstrated herein, we show that DGS-ES is highly scalable, possesses superior wall-clock time, and achieves competitive reward scores to other popular policy gradient and ES approaches.

References in corpus (17)

• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Noisy Networks for Exploration
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Large-Scale Evolution of Image Classifiers
• Recurrent World Models Facilitate Policy Evolution
• Parameter Space Noise for Exploration
• Count-Based Exploration with Neural Density Models
• Hindsight Experience Replay
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks
• Guided evolutionary strategies: Augmenting random search with surrogate gradients
• Provably Robust Blackbox Optimization for Reinforcement Learning