Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents

Abstract

Evolution strategies (ES) are a family of black-box optimization algorithms able to train deep neural networks roughly as well as Q-learning and policy gradient methods on challenging deep reinforcement learning (RL) problems, but are much faster (e.g. hours vs. days) because they parallelize better. However, many RL problems require directed exploration because they have reward functions that are sparse or deceptive (i.e. contain local optima), and it is unknown how to encourage such exploration with ES. Here we show that algorithms that have been invented to promote directed exploration in small-scale evolved neural networks via populations of exploring agents, specifically novelty search (NS) and quality diversity (QD) algorithms, can be hybridized with ES to improve its performance on sparse or deceptive deep RL tasks, while retaining scalability. Our experiments confirm that the resultant new algorithms, NS-ES and two QD algorithms, NSR-ES and NSRA-ES, avoid local optima encountered by ES to achieve higher performance on Atari and simulated robots learning to walk around a deceptive trap. This paper thus introduces a family of fast, scalable algorithms for reinforcement learning that are capable of directed exploration. It also adds this new family of exploration algorithms to the RL toolbox and raises the interesting possibility that analogous algorithms with multiple simultaneous paths of exploration might also combine well with existing RL algorithms outside ES.

References in corpus (4)

• Robots that can adapt like animals
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Illuminating search spaces by mapping elites
• Parameter Space Noise for Exploration

Cited by in corpus (17)

• Exploration in Deep Reinforcement Learning: A Survey
• State Representation Learning for Control: An Overview
• First return, then explore
• Variational Quantum Reinforcement Learning via Evolutionary Optimization
• Evolutionary Reinforcement Learning: A Survey
• Discovering the Elite Hypervolume by Leveraging Interspecies Correlation
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• EA-LSTM: Evolutionary Attention-based LSTM for Time Series Prediction
• Deep Surrogate Assisted MAP-Elites for Automated Hearthstone Deckbuilding
• Guiding Neuroevolution with Structural Objectives
• Policy Search with Rare Significant Events: Choosing the Right Partner to Cooperate with
• Evolutionary Training and Abstraction Yields Algorithmic Generalization of Neural Computers
• Learning Diverse Policies with Soft Self-Generated Guidance
• Effects of Different Optimization Formulations in Evolutionary Reinforcement Learning on Diverse Behavior Generation
• Curiosity creates Diversity in Policy Search
• Learning Representations and Agents for Information Retrieval
• Evolutionary Self-Replication as a Mechanism for Producing Artificial Intelligence