Structured Evolution with Compact Architectures for Scalable Policy Optimization

Abstract

We present a new method of blackbox optimization via gradient approximation with the use of structured random orthogonal matrices, providing more accurate estimators than baselines and with provable theoretical guarantees. We show that this algorithm can be successfully applied to learn better quality compact policies than those using standard gradient estimation techniques. The compact policies we learn have several advantages over unstructured ones, including faster training algorithms and faster inference. These benefits are important when the policy is deployed on real hardware with limited resources. Further, compact policies provide more scalable architectures for derivative-free optimization (DFO) in high-dimensional spaces. We show that most robotics tasks from the OpenAI Gym can be solved using neural networks with less than 300 parameters, with almost linear time complexity of the inference phase, with up to 13x fewer parameters relative to the Evolution Strategies (ES) algorithm introduced by Salimans et al. (2017). We do not need heuristics such as fitness shaping to learn good quality policies, resulting in a simple and theoretically motivated training mechanism.

Cited by in corpus (19)

• Gradientless Descent: High-Dimensional Zeroth-Order Optimization
• MLGO: a Machine Learning Guided Compiler Optimizations Framework
• BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations
• Robotic Table Tennis: A Case Study into a High Speed Learning System
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• Zeroth-Order Algorithms for Smooth Saddle-Point Problems
• Improving Gradient Estimation in Evolutionary Strategies With Past Descent Directions
• Improving Neural Network Training in Low Dimensional Random Bases
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• Nonconvex Zeroth-Order Stochastic ADMM Methods with Lower Function Query Complexity
• Understanding and Training Deep Diagonal Circulant Neural Networks
• Structured Monte Carlo Sampling for Nonisotropic Distributions via Determinantal Point Processes
• A Scalable Gradient-Free Method for Bayesian Experimental Design with Implicit Models
• A stochastic subspace approach to gradient-free optimization in high dimensions
• Distributionally Constrained Black-Box Stochastic Gradient Estimation and Optimization
• Improved Complexities for Stochastic Conditional Gradient Methods under Interpolation-like Conditions
• Neural Auto-Curricula
• An Accelerated Variance-Reduced Conditional Gradient Sliding Algorithm for First-order and Zeroth-order Optimization
• A Hybrid Gradient Method to Designing Bayesian Experiments for Implicit Models