Evolutionary Reinforcement Learning: A Survey
arXiv:2303.04150 · doi:10.34133/icomputing.0025
Abstract
Reinforcement learning (RL) is a machine learning approach that trains agents to maximize cumulative rewards through interactions with environments. The integration of RL with deep learning has recently resulted in impressive achievements in a wide range of challenging tasks, including board games, arcade games, and robot control. Despite these successes, there remain several crucial challenges, including brittle convergence properties caused by sensitive hyperparameters, difficulties in temporal credit assignment with long time horizons and sparse rewards, a lack of diverse exploration, especially in continuous search space scenarios, difficulties in credit assignment in multi-agent reinforcement learning, and conflicting objectives for rewards. Evolutionary computation (EC), which maintains a population of learning agents, has demonstrated promising performance in addressing these limitations. This article presents a comprehensive survey of state-of-the-art methods for integrating EC into RL, referred to as evolutionary reinforcement learning (EvoRL). We categorize EvoRL methods according to key research fields in RL, including hyperparameter optimization, policy search, exploration, reward shaping, meta-RL, and multi-objective RL. We then discuss future research directions in terms of efficient methods, benchmarks, and scalable platforms. This survey serves as a resource for researchers and practitioners interested in the field of EvoRL, highlighting the important challenges and opportunities for future research. With the help of this survey, researchers and practitioners can develop more efficient methods and tailored benchmarks for EvoRL, further advancing this promising cross-disciplinary research field.
References in corpus (37)
- Practical Bayesian Optimization of Machine Learning Algorithms
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Robots that can adapt like animals
- Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
- Deep Reinforcement Learning: An Overview
- A Survey of Multi-Objective Sequential Decision-Making
- Competitive Coevolution through Evolutionary Complexification
- Population Based Training of Neural Networks
- Unifying Count-Based Exploration and Intrinsic Motivation
- First return, then explore
- Go-Explore: a New Approach for Hard-Exploration Problems
- A Survey of Stochastic Simulation and Optimization Methods in Signal Processing
- Evolved Policy Gradients
- Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
- Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
- Proximal Distilled Evolutionary Reinforcement Learning
- Autonomous skill discovery with Quality-Diversity and Unsupervised Descriptors
- Scaling MAP-Elites to Deep Neuroevolution
- Derivative-Free Reinforcement Learning: A Review
- A Self-Tuning Actor-Critic Algorithm
- Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution
- Quality Diversity Through Surprise
- Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination
- Instance Weighted Incremental Evolution Strategies for Reinforcement Learning in Dynamic Environments
- A Surrogate-Assisted Controller for Expensive Evolutionary Reinforcement Learning
- Evolutionary Reinforcement Learning via Cooperative Coevolutionary Negatively Correlated Search
- EvoJAX: Hardware-Accelerated Neuroevolution
- Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots
- Few-shot Quality-Diversity Optimization
- Safe Crossover of Neural Networks Through Neuron Alignment
- Sparse Reward Exploration via Novelty Search and Emitters
- Hybrid Encoding For Generating Large Scale Game Level Patterns With Local Variations
- Novelty Search in Representational Space for Sample Efficient Exploration
- An Efficient Asynchronous Method for Integrating Evolutionary and Gradient-based Policy Search
- Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey
- Behavior-based Neuroevolutionary Training in Reinforcement Learning
- Effects of Different Optimization Formulations in Evolutionary Reinforcement Learning on Diverse Behavior Generation