Few-shot Quality-Diversity Optimization

Abstract

In the past few years, a considerable amount of research has been dedicated to the exploitation of previous learning experiences and the design of Few-shot and Meta Learning approaches, in problem domains ranging from Computer Vision to Reinforcement Learning based control. A notable exception, where to the best of our knowledge, little to no effort has been made in this direction is Quality-Diversity (QD) optimization. QD methods have been shown to be effective tools in dealing with deceptive minima and sparse rewards in Reinforcement Learning. However, they remain costly due to their reliance on inherently sample inefficient evolutionary processes. We show that, given examples from a task distribution, information about the paths taken by optimization in parameter space can be leveraged to build a prior population, which when used to initialize QD methods in unseen environments, allows for few-shot adaptation. Our proposed method does not require backpropagation. It is simple to implement and scale, and furthermore, it is agnostic to the underlying models that are being trained. Experiments carried in both sparse and dense reward settings using robotic manipulation and navigation benchmarks show that it considerably reduces the number of generations that are required for QD optimization in these environments.

arXiv commentAccepted for publication in the IEEE Robotics and Automation Letters (RA-L) journal

References in corpus (18)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Robots that can adapt like animals
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Illuminating search spaces by mapping elites
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Meta-Gradient Reinforcement Learning
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Procedural Content Generation through Quality Diversity
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space
• Model-based Reinforcement Learning: A Survey
• Discovering Reinforcement Learning Algorithms
• Scaling MAP-Elites to Deep Neuroevolution
• ES-MAML: Simple Hessian-Free Meta Learning
• Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
• From exploration to control: learning object manipulation skills through novelty search and local adaptation
• Sparse Reward Exploration via Novelty Search and Emitters

Cited by in corpus (2)

• Evolutionary Reinforcement Learning: A Survey
• Adaptive Asynchronous Control Using Meta-learned Neural Ordinary Differential Equations