CEM-RL: Combining evolutionary and gradient-based methods for policy search

Abstract

Deep neuroevolution and deep reinforcement learning (deep RL) algorithms are two popular approaches to policy search. The former is widely applicable and rather stable, but suffers from low sample efficiency. By contrast, the latter is more sample efficient, but the most sample efficient variants are also rather unstable and highly sensitive to hyper-parameter setting. So far, these families of methods have mostly been compared as competing tools. However, an emerging approach consists in combining them so as to get the best of both worlds. Two previously existing combinations use either an ad hoc evolutionary algorithm or a goal exploration process together with the Deep Deterministic Policy Gradient (DDPG) algorithm, a sample efficient off-policy deep RL algorithm. In this paper, we propose a different combination scheme using the simple cross-entropy method (CEM) and Twin Delayed Deep Deterministic policy gradient (td3), another off-policy deep RL algorithm which improves over ddpg. We evaluate the resulting method, cem-rl, on a set of benchmarks classically used in deep RL. We show that cem-rl benefits from several advantages over its competitors and offers a satisfactory trade-off between performance and sample efficiency.

arXiv commentaccepted at ICLR 2019

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Neural Architecture Search with Reinforcement Learning
• Addressing Function Approximation Error in Actor-Critic Methods
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Deep Exploration via Bootstrapped DQN
• Learning to reinforcement learn
• Parameter Space Noise for Exploration
• Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Path Integral Policy Improvement with Covariance Matrix Adaptation
• Evolved Policy Gradients
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• Importance mixing: Improving sample reuse in evolutionary policy search methods

Cited by in corpus (25)

• Population-Based Black-Box Optimization for Biological Sequence Design
• Proximal Distilled Evolutionary Reinforcement Learning
• Automated Machine Learning on Graphs: A Survey
• Effective Diversity in Population Based Reinforcement Learning
• Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
• Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching
• Reinforcement Learning Driven Heuristic Optimization
• Q-Learning for Continuous Actions with Cross-Entropy Guided Policies
• Model-Predictive Control via Cross-Entropy and Gradient-Based Optimization
• Building Human-like Communicative Intelligence: A Grounded Perspective
• Time Efficiency in Optimization with a Bayesian-Evolutionary Algorithm
• A One-bit, Comparison-Based Gradient Estimator
• Maximum Mutation Reinforcement Learning for Scalable Control
• Smooth Exploration for Robotic Reinforcement Learning
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• GRAC: Self-Guided and Self-Regularized Actor-Critic
• Symbolic Regression via Neural-Guided Genetic Programming Population Seeding
• Recruitment-imitation Mechanism for Evolutionary Reinforcement Learning
• EVO-RL: Evolutionary-Driven Reinforcement Learning
• Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy
• MLE-guided parameter search for task loss minimization in neural sequence modeling
• Regularly Updated Deterministic Policy Gradient Algorithm
• Learning in Sparse Rewards settings through Quality-Diversity algorithms
• Learn a Prior for RHEA for Better Online Planning
• FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control