Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
arXiv:1712.06560
Abstract
Evolution strategies (ES) are a family of black-box optimization algorithms able to train deep neural networks roughly as well as Q-learning and policy gradient methods on challenging deep reinforcement learning (RL) problems, but are much faster (e.g. hours vs. days) because they parallelize better. However, many RL problems require directed exploration because they have reward functions that are sparse or deceptive (i.e. contain local optima), and it is unknown how to encourage such exploration with ES. Here we show that algorithms that have been invented to promote directed exploration in small-scale evolved neural networks via populations of exploring agents, specifically novelty search (NS) and quality diversity (QD) algorithms, can be hybridized with ES to improve its performance on sparse or deceptive deep RL tasks, while retaining scalability. Our experiments confirm that the resultant new algorithms, NS-ES and two QD algorithms, NSR-ES and NSRA-ES, avoid local optima encountered by ES to achieve higher performance on Atari and simulated robots learning to walk around a deceptive trap. This paper thus introduces a family of fast, scalable algorithms for reinforcement learning that are capable of directed exploration. It also adds this new family of exploration algorithms to the RL toolbox and raises the interesting possibility that analogous algorithms with multiple simultaneous paths of exploration might also combine well with existing RL algorithms outside ES.
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- Overcoming catastrophic forgetting in neural networks
- Playing Atari with Deep Reinforcement Learning
- Conditional Image Generation with PixelCNN Decoders
- Improved Techniques for Training GANs
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Robots that can adapt like animals
- Noisy Networks for Exploration
- Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
- VIME: Variational Information Maximizing Exploration
- Illuminating search spaces by mapping elites
- Parameter Space Noise for Exploration
- Count-Based Exploration with Neural Density Models
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
- Unifying Count-Based Exploration and Intrinsic Motivation
- Stein Variational Policy Gradient
- Diffusion-based neuromodulation can eliminate catastrophic forgetting in simple neural networks
- Systematic Derivation of Behaviour Characterisations in Evolutionary Robotics
Cited by in corpus (17)
- Exploration in Deep Reinforcement Learning: A Survey
- State Representation Learning for Control: An Overview
- First return, then explore
- Variational Quantum Reinforcement Learning via Evolutionary Optimization
- Evolutionary Reinforcement Learning: A Survey
- Discovering the Elite Hypervolume by Leveraging Interspecies Correlation
- An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
- EA-LSTM: Evolutionary Attention-based LSTM for Time Series Prediction
- Deep Surrogate Assisted MAP-Elites for Automated Hearthstone Deckbuilding
- Guiding Neuroevolution with Structural Objectives
- Policy Search with Rare Significant Events: Choosing the Right Partner to Cooperate with
- Evolutionary Training and Abstraction Yields Algorithmic Generalization of Neural Computers
- Learning Diverse Policies with Soft Self-Generated Guidance
- Effects of Different Optimization Formulations in Evolutionary Reinforcement Learning on Diverse Behavior Generation
- Curiosity creates Diversity in Policy Search
- Learning Representations and Agents for Information Retrieval
- Evolutionary Self-Replication as a Mechanism for Producing Artificial Intelligence