Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires
arXiv:2109.08522 · doi:10.1109/ICRA46639.2022.9811559
Abstract
Quality-Diversity (QD) algorithms are powerful exploration algorithms that allow robots to discover large repertoires of diverse and high-performing skills. However, QD algorithms are sample inefficient and require millions of evaluations. In this paper, we propose Dynamics-Aware Quality-Diversity (DA-QD), a framework to improve the sample efficiency of QD algorithms through the use of dynamics models. We also show how DA-QD can then be used for continual acquisition of new skill repertoires. To do so, we incrementally train a deep dynamics model from experience obtained when performing skill discovery using QD. We can then perform QD exploration in imagination with an imagined skill repertoire. We evaluate our approach on three robotic experiments. First, our experiments show DA-QD is 20 times more sample efficient than existing QD approaches for skill discovery. Second, we demonstrate learning an entirely new skill repertoire in imagination to perform zero-shot learning. Finally, we show how DA-QD is useful and effective for solving a long horizon navigation task and for damage adaptation in the real world. Videos and source code are available at: https://sites.google.com/view/da-qd.
References in corpus (22)
- Adam: A Method for Stochastic Optimization
- Robots that can adapt like animals
- Illuminating search spaces by mapping elites
- Learning Latent Dynamics for Planning from Pixels
- Model-Ensemble Trust-Region Policy Optimization
- Benchmarking Model-Based Reinforcement Learning
- First return, then explore
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
- Variational Intrinsic Control
- Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space
- Model-Based Reinforcement Learning via Meta-Policy Optimization
- Reset-free Trial-and-Error Learning for Robot Damage Recovery
- Discovering the Elite Hypervolume by Leveraging Interspecies Correlation
- Dynamics-Aware Unsupervised Discovery of Skills
- Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration
- A Game Theoretic Framework for Model Based Reinforcement Learning
- Scaling MAP-Elites to Deep Neuroevolution
- AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
- Adaptive Prior Selection for Repertoire-based Online Adaptation in Robotics
- Multi-Emitter MAP-Elites: Improving quality, diversity and convergence speed with heterogeneous sets of emitters
- Fast Task Inference with Variational Intrinsic Successor Features
- Using Centroidal Voronoi Tessellations to Scale Up the Multi-dimensional Archive of Phenotypic Elites Algorithm
Cited by in corpus (4)
- Deep Surrogate Assisted MAP-Elites for Automated Hearthstone Deckbuilding
- Adaptive Asynchronous Control Using Meta-learned Neural Ordinary Differential Equations
- Improving the Data Efficiency of Multi-Objective Quality-Diversity through Gradient Assistance and Crowding Exploration
- Quality-Diversity Optimisation on a Physical Robot Through Dynamics-Aware and Reset-Free Learning