Scaling MAP-Elites to Deep Neuroevolution

Abstract

Quality-Diversity (QD) algorithms, and MAP-Elites (ME) in particular, have proven very useful for a broad range of applications including enabling real robots to recover quickly from joint damage, solving strongly deceptive maze tasks or evolving robot morphologies to discover new gaits. However, present implementations of MAP-Elites and other QD algorithms seem to be limited to low-dimensional controllers with far fewer parameters than modern deep neural network models. In this paper, we propose to leverage the efficiency of Evolution Strategies (ES) to scale MAP-Elites to high-dimensional controllers parameterized by large neural networks. We design and evaluate a new hybrid algorithm called MAP-Elites with Evolution Strategies (ME-ES) for post-damage recovery in a difficult high-dimensional control task where traditional ME fails. Additionally, we show that ME-ES performs efficient exploration, on par with state-of-the-art exploration algorithms in high-dimensional control tasks with strongly deceptive rewards.

Accepted to GECCO 2020

References in corpus (2)

• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Reset-free Trial-and-Error Learning for Robot Damage Recovery

Cited by in corpus (15)

• Evolutionary Reinforcement Learning: A Survey
• Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
• Multi-Objective Quality Diversity Optimization
• Can Transfer Neuroevolution Tractably Solve Your Differential Equations?
• Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution
• Deep Surrogate Assisted MAP-Elites for Automated Hearthstone Deckbuilding
• Empirical analysis of PGA-MAP-Elites for Neuroevolution in Uncertain Domains
• Evolving and Merging Hebbian Learning Rules: Increasing Generalization by Decreasing the Number of Rules
• Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires
• Evolving the Behavior of Machines: From Micro to Macroevolution
• Few-shot Quality-Diversity Optimization
• Learning to Walk Autonomously via Reset-Free Quality-Diversity
• Don't Bet on Luck Alone: Enhancing Behavioral Reproducibility of Quality-Diversity Solutions in Uncertain Domains
• The Quality-Diversity Transformer: Generating Behavior-Conditioned Trajectories with Decision Transformers
• Enhancing MAP-Elites with Multiple Parallel Evolution Strategies