Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
arXiv:2006.08505 · doi:10.1145/3512290.3528845
Abstract
A fascinating aspect of nature lies in its ability to produce a large and diverse collection of organisms that are all high-performing in their niche. By contrast, most AI algorithms focus on finding a single efficient solution to a given problem. Aiming for diversity in addition to performance is a convenient way to deal with the exploration-exploitation trade-off that plays a central role in learning. It also allows for increased robustness when the returned collection contains several working solutions to the considered problem, making it well-suited for real applications such as robotics. Quality-Diversity (QD) methods are evolutionary algorithms designed for this purpose. This paper proposes a novel algorithm, QDPG, which combines the strength of Policy Gradient algorithms and Quality Diversity approaches to produce a collection of diverse and high-performing neural policies in continuous control environments. The main contribution of this work is the introduction of a Diversity Policy Gradient (DPG) that exploits information at the time-step level to drive policies towards more diversity in a sample-efficient manner. Specifically, QDPG selects neural controllers from a MAP-Elites grid and uses two gradient-based mutation operators to improve both quality and diversity. Our results demonstrate that QDPG is significantly more sample-efficient than its evolutionary competitors.
Add several baselines (Policy Gradient assisted MAP Elites, DIAYN, AGAC) Change writing to take the point of view of the evo community Change style, writing, explanation, figures
References in corpus (22)
- Continuous control with deep reinforcement learning
- Addressing Function Approximation Error in Actor-Critic Methods
- Soft Actor-Critic Algorithms and Applications
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Robots that can adapt like animals
- Illuminating search spaces by mapping elites
- Exploration by Random Network Distillation
- Go-Explore: a New Approach for Hard-Exploration Problems
- Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
- CEM-RL: Combining evolutionary and gradient-based methods for policy search
- Efficient Exploration via State Marginal Matching
- GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
- Discovering the Elite Hypervolume by Leveraging Interspecies Correlation
- Evolution-Guided Policy Gradient in Reinforcement Learning
- Empowering Quality Diversity in Dungeon Design with Interactive Constrained MAP-Elites
- Scaling MAP-Elites to Deep Neuroevolution
- Planning with Goal-Conditioned Policies
- Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspective
- Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning
- Marginalized State Distribution Entropy Regularization in Policy Optimization
- Population-Guided Parallel Policy Search for Reinforcement Learning