Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution

Abstract

Neuroevolution is an alternative to gradient-based optimisation that has the potential to avoid local minima and allows parallelisation. The main limiting factor is that usually it does not scale well with parameter space dimensionality. Inspired by recent work examining neural network intrinsic dimension and loss landscapes, we hypothesise that there exists a low-dimensional manifold, embedded in the policy network parameter space, around which a high-density of diverse and useful policies are located. This paper proposes a novel method for diversity-based policy search via Neuroevolution, that leverages learned representations of the policy network parameters, by performing policy search in this learned representation space. Our method relies on the Quality-Diversity (QD) framework which provides a principled approach to policy search, and maintains a collection of diverse policies, used as a dataset for learning policy representations. Further, we use the Jacobian of the inverse-mapping function to guide the search in the representation space. This ensures that the generated samples remain in the high-density regions, after mapping back to the original space. Finally, we evaluate our contributions on four continuous-control tasks in simulated environments, and compare to diversity-based baselines.

arXiv commentAccepted as a full paper at Genetic and Evolutionary Computation Conference, GECCO 2021. arXiv admin note: substantial text overlap with arXiv:2012.08676

References in corpus (17)

• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Robots that can adapt like animals
• Predicting Parameters in Deep Learning
• OpenAI Gym
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Dynamic Filter Networks
• Illuminating search spaces by mapping elites
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Essentially No Barriers in Neural Network Energy Landscape
• Using Fast Weights to Attend to the Recent Past
• Procedural Content Generation through Quality Diversity
• One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
• Discovering the Elite Hypervolume by Leveraging Interspecies Correlation
• Scaling MAP-Elites to Deep Neuroevolution
• A Generative Process for Sampling Contractive Auto-Encoders
• Biologically inspired architectures for sample-efficient deep reinforcement learning

Cited by in corpus (9)

• Evolutionary Reinforcement Learning: A Survey
• Language Model Crossover: Variation through Few-Shot Prompting
• Differentiable Quality Diversity
• Deep Surrogate Assisted MAP-Elites for Automated Hearthstone Deckbuilding
• Empirical analysis of PGA-MAP-Elites for Neuroevolution in Uncertain Domains
• Generative Design through Quality-Diversity Data Synthesis and Language Models
• The Quality-Diversity Transformer: Generating Behavior-Conditioned Trajectories with Decision Transformers
• Evolutionary Diversity Optimisation in Constructing Satisfying Assignments
• On Evolvability and Behavior Landscapes in Neuroevolutionary Divergent Search