Empirical analysis of PGA-MAP-Elites for Neuroevolution in Uncertain Domains
arXiv:2210.13156 · doi:10.1145/3577203
Abstract
Quality-Diversity algorithms, among which MAP-Elites, have emerged as powerful alternatives to performance-only optimisation approaches as they enable generating collections of diverse and high-performing solutions to an optimisation problem. However, they are often limited to low-dimensional search spaces and deterministic environments. The recently introduced Policy Gradient Assisted MAP-Elites (PGA-MAP-Elites) algorithm overcomes this limitation by pairing the traditional Genetic operator of MAP-Elites with a gradient-based operator inspired by Deep Reinforcement Learning. This new operator guides mutations toward high-performing solutions using policy-gradients. In this work, we propose an in-depth study of PGA-MAP-Elites. We demonstrate the benefits of policy-gradients on the performance of the algorithm and the reproducibility of the generated solutions when considering uncertain domains. We first prove that PGA-MAP-Elites is highly performant in both deterministic and uncertain high-dimensional environments, decorrelating the two challenges it tackles. Secondly, we show that in addition to outperforming all the considered baselines, the collections of solutions generated by PGA-MAP-Elites are highly reproducible in uncertain environments, approaching the reproducibility of solutions found by Quality-Diversity approaches built specifically for uncertain applications. Finally, we propose an ablation and in-depth analysis of the dynamic of the policy-gradients-based variation. We demonstrate that the policy-gradient variation operator is determinant to guarantee the performance of PGA-MAP-Elites but is only essential during the early stage of the process, where it finds high-performing regions of the search space.
arXiv commentsubmitted to Transactions on Evolutionary Learning and Optimization
References in corpus (16)
- A Brief Survey of Deep Reinforcement Learning
- Robots that can adapt like animals
- The CMA Evolution Strategy: A Tutorial
- Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
- Procedural Content Generation through Quality Diversity
- Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space
- Discovering the Elite Hypervolume by Leveraging Interspecies Correlation
- Scaling MAP-Elites to Deep Neuroevolution
- Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
- Fast and stable MAP-Elites in noisy domains using deep grids
- Hierarchical Behavioral Repertoires with Unsupervised Descriptors
- Discovering Representations for Black-box Optimization
- Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution
- Multi-Emitter MAP-Elites: Improving quality, diversity and convergence speed with heterogeneous sets of emitters
- Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learning
- pyribs: A Bare-Bones Python Library for Quality Diversity Optimization