Variance Reduction for Evolution Strategies via Structured Control Variates

Abstract

Evolution Strategies (ES) are a powerful class of blackbox optimization techniques that recently became a competitive alternative to state-of-the-art policy gradient (PG) algorithms for reinforcement learning (RL). We propose a new method for improving accuracy of the ES algorithms, that as opposed to recent approaches utilizing only Monte Carlo structure of the gradient estimator, takes advantage of the underlying MDP structure to reduce the variance. We observe that the gradient estimator of the ES objective can be alternatively computed using reparametrization and PG estimators, which leads to new control variate techniques for gradient estimation in ES optimization. We provide theoretical insights and show through extensive experiments that this RL-specific variance reduction approach outperforms general purpose variance reduction methods.

Accepted to AISTATS (International Conference on Artificial Intelligence and Statistics), 2020 in Palermo, Italy

References in corpus (7)

• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• DeepMind Control Suite
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• Variational Bayesian Inference with Stochastic Search
• Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
• Action-depedent Control Variates for Policy Optimization via Stein's Identity

Cited by in corpus (2)

• Provably Robust Blackbox Optimization for Reinforcement Learning
• Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies