Competitiveness of MAP-Elites against Proximal Policy Optimization on locomotion tasks in deterministic simulations

Abstract

The increasing importance of robots and automation creates a demand for learnable controllers which can be obtained through various approaches such as Evolutionary Algorithms (EAs) or Reinforcement Learning (RL). Unfortunately, these two families of algorithms have mainly developed independently and there are only a few works comparing modern EAs with deep RL algorithms. We show that Multidimensional Archive of Phenotypic Elites (MAP-Elites), which is a modern EA, can deliver better-performing solutions than one of the state-of-the-art RL methods, Proximal Policy Optimization (PPO) in the generation of locomotion controllers for a simulated hexapod robot. Additionally, extensive hyper-parameter tuning shows that MAP-Elites displays greater robustness across seeds and hyper-parameter sets. Generally, this paper demonstrates that EAs combined with modern computational resources display promising characteristics and have the potential to contribute to the state-of-the-art in controller learning.

arXiv commentQuality-Diversity optimization, Reinforcement Learning, Proximal Policy Optimization, MAP-Elites

References in corpus (12)

• Deep Learning in Neural Networks: An Overview
• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Robots that can adapt like animals
• Emergence of Locomotion Behaviours in Rich Environments
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Illuminating search spaces by mapping elites
• Adaptive Prior Selection for Repertoire-based Online Adaptation in Robotics