Provable Guarantees for Gradient-Based Meta-Learning

Abstract

We study the problem of meta-learning through the lens of online convex optimization, developing a meta-algorithm bridging the gap between popular gradient-based meta-learning and classical regularization-based multi-task transfer methods. Our method is the first to simultaneously satisfy good sample efficiency guarantees in the convex setting, with generalization bounds that improve with task-similarity, while also being computationally scalable to modern deep learning architectures and the many-task setting. Despite its simplicity, the algorithm matches, up to a constant factor, a lower bound on the performance of any such parameter-transfer method under natural task similarity assumptions. We use experiments in both convex and deep learning settings to verify and demonstrate the applicability of our theory.

ICML 2019

References in corpus (9)

• Prototypical Networks for Few-shot Learning
• Theoretical Models of Learning to Learn
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Online Learning with Predictable Sequences
• A PAC-Bayesian bound for Lifelong Learning
• Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory
• Auto-Meta: Automated Gradient Based Meta Learner Search

Cited by in corpus (15)

• On the Theory of Transfer Learning: The Importance of Task Diversity
• Solving Stochastic Compositional Optimization is Nearly as Easy as Solving Stochastic Optimization
• Provable Meta-Learning of Linear Representations
• Differentiable Meta-learning Model for Few-shot Semantic Segmentation
• On Localized Discrepancy for Domain Adaptation
• Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayes
• Theoretical bounds on estimation error for meta-learning
• Variable-Shot Adaptation for Online Meta-Learning
• System Identification via Meta-Learning in Linear Time-Varying Environments
• Meta R-CNN : Towards General Solver for Instance-level Few-shot Learning
• Meta-Learning Conjugate Priors for Few-Shot Bayesian Optimization
• Online Bilevel Optimization: Regret Analysis of Online Alternating Gradient Methods
• Compositional federated learning: Applications in distributionally robust averaging and meta learning
• Meta-Learning Guarantees for Online Receding Horizon Learning Control
• A Meta-Learning Control Algorithm with Provable Finite-Time Guarantees