How to Train Your MAML to Excel in Few-Shot Classification

Abstract

Model-agnostic meta-learning (MAML) is arguably one of the most popular meta-learning algorithms nowadays. Nevertheless, its performance on few-shot classification is far behind many recent algorithms dedicated to the problem. In this paper, we point out several key facets of how to train MAML to excel in few-shot classification. First, we find that MAML needs a large number of gradient steps in its inner loop update, which contradicts its common usage in few-shot classification. Second, we find that MAML is sensitive to the class label assignments during meta-testing. Concretely, MAML meta-trains the initialization of an $N$-way classifier. These $N$ ways, during meta-testing, then have "$N!$" different permutations to be paired with a few-shot task of $N$ novel classes. We find that these permutations lead to a huge variance of accuracy, making MAML unstable in few-shot classification. Third, we investigate several approaches to make MAML permutation-invariant, among which meta-training a single vector to initialize all the $N$ weight vectors in the classification head performs the best. On benchmark datasets like MiniImageNet and TieredImageNet, our approach, which we name UNICORN-MAML, performs on a par with or even outperforms many recent few-shot classification algorithms, without sacrificing MAML's simplicity.

arXiv commentAccepted to International Conference on Learning Representations 2022 (ICLR 2022)

References in corpus (25)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Neural Architecture Search: A Survey
• YFCC100M: The New Data in Multimedia Research
• Theoretical Models of Learning to Learn
• Learning to Reweight Examples for Robust Deep Learning
• On First-Order Meta-Learning Algorithms
• DropBlock: A regularization method for convolutional networks
• Meta-Learning with Implicit Gradients
• Neural Optimizer Search with Reinforcement Learning
• Meta-Learning: A Survey
• Unsupervised Learning via Meta-Learning
• Hierarchically Structured Meta-learning
• Learned Optimizers that Scale and Generalize
• The Benefit of Multitask Representation Learning
• Meta-Learning Update Rules for Unsupervised Representation Learning
• Attentive Recurrent Comparators
• Feature-Critic Networks for Heterogeneous Domain Generalization
• Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation
• Meta-Learning Transferable Active Learning Policies by Deep Reinforcement Learning
• Learning Algorithms for Active Learning
• Meta-Learning Requires Meta-Augmentation
• Supervising Unsupervised Learning
• Embedding Adaptation is Still Needed for Few-Shot Learning
• Meta-learning with negative learning rates