Unsupervised Meta-Learning for Reinforcement Learning

Abstract

Meta-learning algorithms use past experience to learn to quickly solve new tasks. In the context of reinforcement learning, meta-learning algorithms acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks. The performance of meta-learning algorithms depends on the tasks available for meta-training: in the same way that supervised learning generalizes best to test points drawn from the same distribution as the training points, meta-learning methods generalize best to tasks from the same distribution as the meta-training tasks. In effect, meta-reinforcement learning offloads the design burden from algorithm design to task design. If we can automate the process of task design as well, we can devise a meta-learning algorithm that is truly automated. In this work, we take a step in this direction, proposing a family of unsupervised meta-learning algorithms for reinforcement learning. We motivate and describe a general recipe for unsupervised meta-reinforcement learning, and present an instantiation of this approach. Our conceptual and theoretical contributions consist of formulating the unsupervised meta-reinforcement learning problem and describing how task proposals based on mutual information can be used to train optimal meta-learners. Our experimental results indicate that unsupervised meta-reinforcement learning effectively acquires accelerated reinforcement learning procedures without the need for manual task design and these procedures exceed the performance of learning from scratch.

arXiv commentFirst two authors contributed equally

References in corpus (39)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Theoretical Models of Learning to Learn
• A Simple Neural Attentive Meta-Learner
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• VIME: Variational Information Maximizing Exploration
• Deep Exploration via Bootstrapped DQN
• Learning to reinforcement learn
• Hindsight Experience Replay
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• One-Shot Visual Imitation Learning via Meta-Learning
• Unifying Count-Based Exploration and Intrinsic Motivation
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Data-dependent Initializations of Convolutional Neural Networks
• Variational Intrinsic Control
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Unsupervised Learning via Meta-Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Automatic Goal Generation for Reinforcement Learning Agents
• Evolved Policy Gradients
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• Efficient Exploration via State Marginal Matching
• Meta-Learning Update Rules for Unsupervised Representation Learning
• Universal Planning Networks
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Variational Option Discovery Algorithms
• Dynamics-Aware Unsupervised Discovery of Skills
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Automated Curriculum Learning for Neural Networks
• Assume, Augment and Learn: Unsupervised Few-Shot Meta-Learning via Random Labels and Data Augmentation
• All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
• Unsupervised Curricula for Visual Meta-Reinforcement Learning
• Weight Initialization of Deep Neural Networks(DNNs) using Data Statistics
• Unsupervised Few-shot Learning via Self-supervised Training
• Learning to Transfer: Unsupervised Meta Domain Translation

Cited by in corpus (29)

• Solving Rubik's Cube with a Robot Hand
• Unsupervised Learning via Meta-Learning
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• GeneraLight: Improving Environment Generalization of Traffic Signal Control via Meta Reinforcement Learning
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Model-based Adversarial Meta-Reinforcement Learning
• Concurrent Meta Reinforcement Learning
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• Meta Reinforcement Learning with Task Embedding and Shared Policy
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Probabilistic Active Meta-Learning
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Cross-Trajectory Representation Learning for Zero-Shot Generalization in RL
• Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Skill Discovery of Coordination in Multi-agent Reinforcement Learning
• Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning
• Exploration by Maximizing Rényi Entropy for Reward-Free RL Framework
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks
• System Identification via Meta-Learning in Linear Time-Varying Environments
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey
• Characterizing Policy Divergence for Personalized Meta-Reinforcement Learning
• On the Practical Consistency of Meta-Reinforcement Learning Algorithms
• L2E: Learning to Exploit Your Opponent