On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms

Abstract

We study the convergence of a class of gradient-based Model-Agnostic Meta-Learning (MAML) methods and characterize their overall complexity as well as their best achievable accuracy in terms of gradient norm for nonconvex loss functions. We start with the MAML method and its first-order approximation (FO-MAML) and highlight the challenges that emerge in their analysis. By overcoming these challenges not only we provide the first theoretical guarantees for MAML and FO-MAML in nonconvex settings, but also we answer some of the unanswered questions for the implementation of these algorithms including how to choose their learning rate and the batch size for both tasks and datasets corresponding to tasks. In particular, we show that MAML can find an $\epsilon$-first-order stationary point ($\epsilon$-FOSP) for any positive $\epsilon$ after at most $\mathcal{O}(1/{\epsilon }^2)$ iterations at the expense of requiring second-order information. We also show that FO-MAML which ignores the second-order information required in the update of MAML cannot achieve any small desired level of accuracy, i.e., FO-MAML cannot find an $\epsilon$-FOSP for any $\epsilon >0$. We further propose a new variant of the MAML algorithm called Hessian-free MAML which preserves all theoretical guarantees of MAML, without requiring access to second-order information.

arXiv commentTo appear in the proceedings of the $23^{rd}$ International Conference on Artificial Intelligence and Statistics (AISTATS) 2020

Cited by in corpus (23)

• Personalized Federated Learning with Moreau Envelopes
• From Learning to Meta-Learning: Reduced Training Overhead and Complexity for Communication Systems
• Solving Stochastic Compositional Optimization is Nearly as Easy as Solving Stochastic Optimization
• Inexact-ADMM Based Federated Meta-Learning for Fast and Continual Edge Learning
• Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning
• Task-Robust Model-Agnostic Meta-Learning
• On the Outsized Importance of Learning Rates in Local Update Methods
• Lower Bounds and Accelerated Algorithms for Bilevel Optimization
• Learned Initializations for Optimizing Coordinate-Based Neural Representations
• Biased Stochastic First-Order Methods for Conditional Stochastic Optimization and Applications in Meta Learning
• Weighted Meta-Learning
• Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning
• How Fine-Tuning Allows for Effective Meta-Learning
• Randomized Stochastic Variance-Reduced Methods for Multi-Task Stochastic Bilevel Optimization
• Bilevel Online Adaptation for Out-of-Domain Human Mesh Reconstruction
• Meta-Learning with Neural Tangent Kernels
• Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis
• How Does the Task Landscape Affect MAML Performance?
• Multi-step Estimation for Gradient-based Meta-learning
• A Primal-Dual Subgradient Approachfor Fair Meta Learning
• Compositional federated learning: Applications in distributionally robust averaging and meta learning
• UAV-assisted Online Machine Learning over Multi-Tiered Networks: A Hierarchical Nested Personalized Federated Learning Approach
• Fair Meta-Learning For Few-Shot Classification