Evolution-Guided Policy Gradient in Reinforcement Learning

Abstract

Deep Reinforcement Learning (DRL) algorithms have been successfully applied to a range of challenging control tasks. However, these methods typically suffer from three core difficulties: temporal credit assignment with sparse rewards, lack of effective exploration, and brittle convergence properties that are extremely sensitive to hyperparameters. Collectively, these challenges severely limit the applicability of these approaches to real-world problems. Evolutionary Algorithms (EAs), a class of black box optimization techniques inspired by natural evolution, are well suited to address each of these three challenges. However, EAs typically suffer from high sample complexity and struggle to solve problems that require optimization of a large number of parameters. In this paper, we introduce Evolutionary Reinforcement Learning (ERL), a hybrid algorithm that leverages the population of an EA to provide diversified data to train an RL agent, and reinserts the RL agent into the EA population periodically to inject gradient information into the EA. ERL inherits EA's ability of temporal credit assignment with a fitness metric, effective exploration with a diverse set of policies, and stability of a population-based approach and complements it with off-policy DRL's ability to leverage gradients for higher sample efficiency and faster learning. Experiments in a range of challenging continuous control benchmarks demonstrate that ERL significantly outperforms prior DRL and EA methods.

arXiv comment32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada

Cited by in corpus (24)

• Proximal Distilled Evolutionary Reinforcement Learning
• Effective Diversity in Population Based Reinforcement Learning
• Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
• Instance Weighted Incremental Evolution Strategies for Reinforcement Learning in Dynamic Environments
• Evolving Rewards to Automate Reinforcement Learning
• Reinforcement Learning Driven Heuristic Optimization
• Evolving Reinforcement Learning Algorithms
• Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
• Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
• Maximum Mutation Reinforcement Learning for Scalable Control
• Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning
• Population-Guided Parallel Policy Search for Reinforcement Learning
• Symbolic Regression via Neural-Guided Genetic Programming Population Seeding
• REIN-2: Giving Birth to Prepared Reinforcement Learning Agents Using Reinforcement Learning Agents
• Sample-Efficient Automated Deep Reinforcement Learning
• Multi-Path Policy Optimization
• Recruitment-imitation Mechanism for Evolutionary Reinforcement Learning
• Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning
• Neural-iLQR: A Learning-Aided Shooting Method for Trajectory Optimization
• Evolutionary Generative Adversarial Networks with Crossover Based Knowledge Distillation
• ConQUR: Mitigating Delusional Bias in Deep Q-learning
• Periodic Intra-Ensemble Knowledge Distillation for Reinforcement Learning
• Regularly Updated Deterministic Policy Gradient Algorithm
• FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control