Guided evolutionary strategies: Augmenting random search with surrogate gradients

Abstract

Many applications in machine learning require optimizing a function whose true gradient is unknown, but where surrogate gradient information (directions that may be correlated with, but not necessarily identical to, the true gradient) is available instead. This arises when an approximate gradient is easier to compute than the full gradient (e.g. in meta-learning or unrolled optimization), or when a true gradient is intractable and is replaced with a surrogate (e.g. in certain reinforcement learning applications, or when using synthetic gradients). We propose Guided Evolutionary Strategies, a method for optimally using surrogate gradient directions along with random search. We define a search distribution for evolutionary strategies that is elongated along a guiding subspace spanned by the surrogate gradients. This allows us to estimate a descent direction which can then be passed to a first-order optimizer. We analytically and numerically characterize the tradeoffs that result from tuning how strongly the search distribution is stretched along the guiding subspace, and we use this to derive a setting of the hyperparameters that works well across problems. Finally, we apply our method to example problems, demonstrating an improvement over both standard evolutionary strategies and first-order methods (that directly follow the surrogate gradient). We provide a demo of Guided ES at https://github.com/brain-research/guided-evolutionary-strategies

Published at ICML 2019

Cited by in corpus (14)

• Improving Black-box Adversarial Attacks with a Transfer-based Prior
• Learning the exchange-correlation functional from nature with fully differentiable density functional theory
• Correspondence between neuroevolution and gradient descent
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Local policy search with Bayesian optimization
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• Improving Gradient Estimation in Evolutionary Strategies With Past Descent Directions
• AdaDGS: An adaptive black-box optimization method with a nonlocal directional Gaussian smoothing gradient
• Structured Monte Carlo Sampling for Nonisotropic Distributions via Determinantal Point Processes
• MLE-guided parameter search for task loss minimization in neural sequence modeling
• Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy
• On the Convergence of Prior-Guided Zeroth-Order Optimization Algorithms
• On the Second-order Convergence Properties of Random Search Methods
• FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control