Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameters

Abstract

Although model-agnostic meta-learning (MAML) is a very successful algorithm in meta-learning practice, it can have high computational cost because it updates all model parameters over both the inner loop of task-specific adaptation and the outer-loop of meta initialization training. A more efficient algorithm ANIL (which refers to almost no inner loop) was proposed recently by Raghu et al. 2019, which adapts only a small subset of parameters in the inner loop and thus has substantially less computational cost than MAML as demonstrated by extensive experiments. However, the theoretical convergence of ANIL has not been studied yet. In this paper, we characterize the convergence rate and the computational complexity for ANIL under two representative inner-loop loss geometries, i.e., strongly-convexity and nonconvexity. Our results show that such a geometric property can significantly affect the overall convergence performance of ANIL. For example, ANIL achieves a faster convergence rate for a strongly-convex inner-loop loss as the number $N$ of inner-loop gradient descent steps increases, but a slower convergence rate for a nonconvex inner-loop loss as $N$ increases. Moreover, our complexity analysis provides a theoretical quantification on the improved efficiency of ANIL over MAML. The experiments on standard few-shot meta-learning benchmarks validate our theoretical findings.

arXiv commentAccepted by NeurIPS 2020

References in corpus (13)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Theoretical Models of Learning to Learn
• One-Shot Visual Imitation Learning via Meta-Learning
• Meta-Learning with Implicit Gradients
• Deep Meta-Learning: Learning to Learn in the Concept Space
• Online Meta-Learning
• Approximation Methods for Bilevel Programming
• Near-Optimal Algorithms for Minimax Optimization
• Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning
• Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization
• Reconciling meta-learning and continual learning with online mixtures of tasks
• History-Gradient Aided Batch Size Adaptation for Variance Reduced Algorithms

Cited by in corpus (11)

• Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning
• Lower Bounds and Accelerated Algorithms for Bilevel Optimization
• How Important is the Train-Validation Split in Meta-Learning?
• Provably Faster Algorithms for Bilevel Optimization
• Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning
• How Fine-Tuning Allows for Effective Meta-Learning
• Sign-MAML: Efficient Model-Agnostic Meta-Learning by SignSGD
• Gradient-based algorithms for multi-objective bi-level optimization
• How Does the Task Landscape Affect MAML Performance?
• Efficient Gradient Approximation Method for Constrained Bilevel Optimization
• Stability and Generalization of Bilevel Programming in Hyperparameter Optimization