Bootstrapped Meta-Learning

Abstract

Meta-learning empowers artificial intelligence to increase its efficiency by learning how to learn. Unlocking this potential involves overcoming a challenging meta-optimisation problem. We propose an algorithm that tackles this problem by letting the meta-learner teach itself. The algorithm first bootstraps a target from the meta-learner, then optimises the meta-learner by minimising the distance to that target under a chosen (pseudo-)metric. Focusing on meta-learning with gradients, we establish conditions that guarantee performance improvements and show that the metric can control meta-optimisation. Meanwhile, the bootstrapping mechanism can extend the effective meta-learning horizon without requiring backpropagation through all updates. We achieve a new state-of-the art for model-free agents on the Atari ALE benchmark and demonstrate that it yields both performance and efficiency gains in multi-task meta-learning. Finally, we explore how bootstrapping opens up new possibilities and find that it can meta-learn efficient exploration in an epsilon-greedy Q-learning agent, without backpropagating through the update rule.

arXiv commentPublished at ICLR 2022. 37 pages, 19 figures, 9 tables

References in corpus (20)

• Distilling the Knowledge in a Neural Network
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Playing Atari with Deep Reinforcement Learning
• Bootstrap your own latent: A new approach to self-supervised Learning
• Trust Region Policy Optimization
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• On First-Order Meta-Learning Algorithms
• Maximum a Posteriori Policy Optimisation
• Fast Context Adaptation via Meta-Learning
• Meta-Gradient Reinforcement Learning
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Discovering Reinforcement Learning Algorithms
• Learning Gradient Descent: Better Generalization and Longer Horizons
• Understanding and correcting pathologies in the training of learned optimizers
• Transferring Knowledge across Learning Processes
• A Self-Tuning Actor-Critic Algorithm
• Mirror Descent Policy Optimization
• Meta-learning curiosity algorithms
• Muesli: Combining Improvements in Policy Optimization