Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning

Abstract

Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement learning (RL) problems. However, ES can be considered a gradient-based algorithm because it performs stochastic gradient descent via an operation similar to a finite-difference approximation of the gradient. That raises the question of whether non-gradient-based evolutionary algorithms can work at DNN scales. Here we demonstrate they can: we evolve the weights of a DNN with a simple, gradient-free, population-based genetic algorithm (GA) and it performs well on hard deep RL problems, including Atari and humanoid locomotion. The Deep GA successfully evolves networks with over four million free parameters, the largest neural networks ever evolved with a traditional evolutionary algorithm. These results (1) expand our sense of the scale at which GAs can operate, (2) suggest intriguingly that in some cases following the gradient is not the best choice for optimizing performance, and (3) make immediately available the multitude of neuroevolution techniques that improve performance. We demonstrate the latter by showing that combining DNNs with novelty search, which encourages exploration on tasks with deceptive or sparse reward functions, can solve a high-dimensional problem on which reward-maximizing algorithms (e.g.\ DQN, A3C, ES, and the GA) fail. Additionally, the Deep GA is faster than ES, A3C, and DQN (it can train Atari in ${\raise.17ex\hbox{$\scriptstyle\sim$}}$4 hours on one desktop or ${\raise.17ex\hbox{$\scriptstyle\sim$}}$1 hour distributed on 720 cores), and enables a state-of-the-art, up to 10,000-fold compact encoding technique.

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Prioritized Experience Replay
• Dueling Network Architectures for Deep Reinforcement Learning
• Improved Techniques for Training GANs
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Robots that can adapt like animals
• Noisy Networks for Exploration
• Self-Normalizing Neural Networks
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Deep Exploration via Bootstrapped DQN
• Illuminating search spaces by mapping elites
• Massively Parallel Methods for Deep Reinforcement Learning
• Parameter Space Noise for Exploration
• A Distributional Perspective on Reinforcement Learning
• Hierarchical Representations for Efficient Architecture Search
• On the saddle point problem for non-convex optimization

Cited by in corpus (121)

• Neural Architecture Search: A Survey
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Neural network models and deep learning - a primer for biologists
• Deep Reinforcement Learning: An Overview
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• Exploration in Deep Reinforcement Learning: A Survey
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Go-Explore: a New Approach for Hard-Exploration Problems
• Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges
• Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks
• Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Review: Deep Learning in Electron Microscopy
• Variational Quantum Reinforcement Learning via Evolutionary Optimization
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Proximal Distilled Evolutionary Reinforcement Learning
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• MaskConnect: Connectivity Learning by Gradient Descent
• ES-MAML: Simple Hessian-Free Meta Learning
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Gradient Descent based Optimization Algorithms for Deep Learning Models Training
• Effective Diversity in Population Based Reinforcement Learning
• The Evolution of Reinforcement Learning in Quantitative Finance: A Survey
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• Autostacker: A Compositional Evolutionary Learning System
• Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution
• Evolving Modular Soft Robots without Explicit Inter-Module Communication using Local Self-Attention
• GenAttack: Practical Black-box Attacks with Gradient-Free Optimization
• GADAM: Genetic-Evolutionary ADAM for Deep Neural Network Optimization
• Evolutionary reinforcement learning of dynamical large deviations
• Behavioural Repertoire via Generative Adversarial Policy Networks
• Learning Self-Imitating Diverse Policies
• Correspondence between neuroevolution and gradient descent
• Empirical analysis of PGA-MAP-Elites for Neuroevolution in Uncertain Domains
• Black-box adversarial attacks using Evolution Strategies
• EvoJAX: Hardware-Accelerated Neuroevolution
• GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
• Balancing Profit, Risk, and Sustainability for Portfolio Management
• Optimal quantum control via genetic algorithms for quantum state engineering in driven-resonator mediated networks
• Playing Atari with Six Neurons
• Global convergence of neuron birth-death dynamics
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Never look back - A modified EnKF method and its application to the training of neural networks without back propagation
• How You Act Tells a Lot: Privacy-Leakage Attack on Deep Reinforcement Learning
• Policy Evaluation Networks
• Inverse Design of a Graphene-Based Quantum Transducer via Neuroevolution
• Qualities, challenges and future of genetic algorithms: a literature review
• Solving the max-3-cut problem using synchronized dissipative networks
• Evolving the Behavior of Machines: From Micro to Macroevolution
• Guiding Neuroevolution with Structural Objectives
• Importance mixing: Improving sample reuse in evolutionary policy search methods
• Neuroevolution of Neural Network Architectures Using CoDeepNEAT and Keras
• Multi-objective Model-based Policy Search for Data-efficient Learning with Sparse Rewards
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• Deep Optimisation: Solving Combinatorial Optimisation Problems using Deep Neural Networks
• Evolvability ES: Scalable and Direct Optimization of Evolvability
• Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
• Smooth Exploration for Robotic Reinforcement Learning
• Evolving Inborn Knowledge For Fast Adaptation in Dynamic POMDP Problems
• DeepMasterPrints: Generating MasterPrints for Dictionary Attacks via Latent Variable Evolution
• Vulnerable road user detection: state-of-the-art and open challenges
• Regenerating Soft Robots through Neural Cellular Automata
• Neuroevolution in Deep Learning: The Role of Neutrality
• Policy Search with Rare Significant Events: Choosing the Right Partner to Cooperate with
• From Complexity to Simplicity: Adaptive ES-Active Subspaces for Blackbox Optimization
• HyperNCA: Growing Developmental Networks with Neural Cellular Automata
• Structured Control Nets for Deep Reinforcement Learning
• Symbolic Regression via Neural-Guided Genetic Programming Population Seeding
• Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods
• Hybrid Self-Attention NEAT: A novel evolutionary approach to improve the NEAT algorithm
• Meta Learning Black-Box Population-Based Optimizers
• Interpretable discovery of new semiconductors with machine learning
• A NEAT Quantum Error Decoder
• Evolutionary Augmentation Policy Optimization for Self-supervised Learning
• Simple Genetic Operators are Universal Approximators of Probability Distributions (and other Advantages of Expressive Encodings)
• RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks
• Learning Guidance Rewards with Trajectory-space Smoothing
• Sample-Efficient Automated Deep Reinforcement Learning
• The Adversarial Resilience Learning Architecture for AI-based Modelling, Exploration, and Operation of Complex Cyber-Physical Systems
• Neural Architecture Search Over a Graph Search Space
• DLOPT: Deep Learning Optimization Library
• Diverse Behavior Is What Game AI Needs: Generating Varied Human-Like Playing Styles Using Evolutionary Multi-Objective Deep Reinforcement Learning
• Learning Fitness Functions for Machine Programming
• Evolutionary Construction of Convolutional Neural Networks
• A Continuous Optimisation Benchmark Suite from Neural Network Regression
• MDEA: Malware Detection with Evolutionary Adversarial Learning
• Evolving Self-supervised Neural Networks: Autonomous Intelligence from Evolved Self-teaching
• Multi-Path Policy Optimization
• Transferable Cost-Aware Security Policy Implementation for Malware Detection Using Deep Reinforcement Learning
• Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy
• Switching Isotropic and Directional Exploration with Parameter Space Noise in Deep Reinforcement Learning
• Generative Design of Hardware-aware DNNs
• Optimizing Deep Neural Networks with Multiple Search Neuroevolution
• Monte-Carlo Tree Search for Policy Optimization
• Learning a Distributed Control Scheme for Demand Flexibility in Thermostatically Controlled Loads
• Reducing catastrophic forgetting when evolving neural networks
• Importance Weighted Evolution Strategies
• Competitiveness of MAP-Elites against Proximal Policy Optimization on locomotion tasks in deterministic simulations
• A coevolutionary approach to deep multi-agent reinforcement learning
• Effects of Different Optimization Formulations in Evolutionary Reinforcement Learning on Diverse Behavior Generation
• Evolutionary Selective Imitation: Interpretable Agents by Imitation Learning Without a Demonstrator
• SURREAL-System: Fully-Integrated Stack for Distributed Deep Reinforcement Learning
• ASCAI: Adaptive Sampling for acquiring Compact AI
• Controllability, Multiplexing, and Transfer Learning in Networks using Evolutionary Learning
• Recurrent Control Nets for Deep Reinforcement Learning
• Assessing and Accelerating Coverage in Deep Reinforcement Learning
• VINE: An Open Source Interactive Data Visualization Tool for Neuroevolution
• General Characterization of Agents by States they Visit
• Genetic-Gated Networks for Deep Reinforcement
• ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spaces
• Supervising Unsupervised Learning with Evolutionary Algorithm in Deep Neural Network
• An Evolutionary Algorithm of Linear complexity: Application to Training of Deep Neural Networks
• Conditional Neural Architecture Search
• Evolving Indoor Navigational Strategies Using Gated Recurrent Units In NEAT
• The Foundations of Deep Learning with a Path Towards General Intelligence
• Population-based Global Optimisation Methods for Learning Long-term Dependencies with RNNs
• Why We Do Not Evolve Software? Analysis of Evolutionary Algorithms
• Reusability and Transferability of Macro Actions for Reinforcement Learning
• Quadratic speedup of global search using a biased crossover of two good solutions