How Does the Task Landscape Affect MAML Performance?
arXiv:2010.14672
Abstract
Model-Agnostic Meta-Learning (MAML) has become increasingly popular for training models that can quickly adapt to new tasks via one or few stochastic gradient descent steps. However, the MAML objective is significantly more difficult to optimize compared to standard non-adaptive learning (NAL), and little is understood about how much MAML improves over NAL in terms of the fast adaptability of their solutions in various scenarios. We analytically address this issue in a linear regression setting consisting of a mixture of easy and hard tasks, where hardness is related to the rate that gradient descent converges on the task. Specifically, we prove that in order for MAML to achieve substantial gain over NAL, (i) there must be some discrepancy in hardness among the tasks, and (ii) the optimal solutions of the hard tasks must be closely packed with the center far from the center of the easy tasks optimal solutions. We also give numerical and analytical results suggesting that these insights apply to two-layer neural networks. Finally, we provide few-shot image classification experiments that support our insights for when MAML should be used and emphasize the importance of training MAML on hard tasks in practice.
References in corpus (24)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
- On First-Order Meta-Learning Algorithms
- Meta-Learning with Implicit Gradients
- A Closer Look at Few-shot Classification
- Recovery Guarantees for One-hidden-layer Neural Networks
- Online Meta-Learning
- Few-Shot Learning via Learning the Representation, Provably
- Provable Meta-Learning of Linear Representations
- On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms
- Guided Meta-Policy Search
- Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning
- Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameters
- Task-Robust Model-Agnostic Meta-Learning
- Adaptive Gradient-Based Meta-Learning Methods
- On the Outsized Importance of Learning Rates in Local Update Methods
- Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks
- Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks
- How Important is the Train-Validation Split in Meta-Learning?
- Expert Training: Task Hardness Aware Meta-Learning for Few-Shot Classification
- Modeling and Optimization Trade-off in Meta-learning
- Guarantees for Tuning the Step Size using a Learning-to-Learn Approach
- A Representation Learning Perspective on the Importance of Train-Validation Splitting in Meta-Learning
- Meta-learning with negative learning rates