Unsupervised Curricula for Visual Meta-Reinforcement Learning

Abstract

In principle, meta-reinforcement learning algorithms leverage experience across many tasks to learn fast reinforcement learning (RL) strategies that transfer to similar tasks. However, current meta-RL approaches rely on manually-defined distributions of training tasks, and hand-crafting these task distributions can be challenging and time-consuming. Can "useful" pre-training tasks be discovered in an unsupervised manner? We develop an unsupervised algorithm for inducing an adaptive meta-training task distribution, i.e. an automatic curriculum, by modeling unsupervised interaction in a visual environment. The task distribution is scaffolded by a parametric density model of the meta-learner's trajectory distribution. We formulate unsupervised meta-RL as information maximization between a latent task variable and the meta-learner's data distribution, and describe a practical instantiation which alternates between integration of recent experience into the task distribution and meta-learning of the updated tasks. Repeating this procedure leads to iterative reorganization such that the curriculum adapts as the meta-learner's data distribution shifts. In particular, we show how discriminative clustering for visual representation can support trajectory-level task acquisition and exploration in domains with pixel observations, avoiding pitfalls of alternatives. In experiments on vision-based navigation and manipulation domains, we show that the algorithm allows for unsupervised meta-learning that transfers to downstream tasks specified by hand-crafted reward functions and serves as pre-training for more efficient supervised meta-learning of test task distributions.

NeurIPS 2019

Cited by in corpus (18)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Unsupervised Meta-Learning for Reinforcement Learning
• Hierarchical Reinforcement Learning By Discovering Intrinsic Options
• BeBold: Exploration Beyond the Boundary of Explored Regions
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Probabilistic Active Meta-Learning
• Meta Automatic Curriculum Learning
• Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Uniform Priors for Data-Efficient Transfer
• Replay-Guided Adversarial Environment Design
• Trying AGAIN instead of Trying Longer: Prior Learning for Automatic Curriculum Learning
• DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning
• Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey