GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms

Abstract

In continuous action domains, standard deep reinforcement learning algorithms like DDPG suffer from inefficient exploration when facing sparse or deceptive reward problems. Conversely, evolutionary and developmental methods focusing on exploration like Novelty Search, Quality-Diversity or Goal Exploration Processes explore more robustly but are less efficient at fine-tuning policies using gradient descent. In this paper, we present the GEP-PG approach, taking the best of both worlds by sequentially combining a Goal Exploration Process and two variants of DDPG. We study the learning performance of these components and their combination on a low dimensional deceptive reward problem and on the larger Half-Cheetah benchmark. We show that DDPG fails on the former and that GEP-PG improves over the best DDPG variant in both environments. Supplementary videos and discussion can be found at http://frama.link/gep_pg, the code at http://github.com/flowersteam/geppg.

accepted at ICML 2018, 14 pages

References in corpus (7)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Parameter Space Noise for Exploration
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic

Cited by in corpus (20)

• Exploration in Deep Reinforcement Learning: A Survey
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
• Computational Theories of Curiosity-Driven Learning
• Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination
• Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research
• The problem with DDPG: understanding failures in deterministic environments with sparse rewards
• Distributionally Robust Reinforcement Learning
• BeBold: Exploration Beyond the Boundary of Explored Regions
• Importance mixing: Improving sample reuse in evolutionary policy search methods
• Robots Learn Increasingly Complex Tasks with Intrinsic Motivation and Automatic Curriculum Learning
• Smooth Exploration for Robotic Reinforcement Learning
• Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewards
• PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Switching Isotropic and Directional Exploration with Parameter Space Noise in Deep Reinforcement Learning
• Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy
• Learning in Sparse Rewards settings through Quality-Diversity algorithms