How to train your MAML

Abstract

The field of few-shot learning has recently seen substantial advancements. Most of these advancements came from casting few-shot learning as a meta-learning problem. Model Agnostic Meta Learning or MAML is currently one of the best approaches for few-shot learning via meta-learning. MAML is simple, elegant and very powerful, however, it has a variety of issues, such as being very sensitive to neural network architectures, often leading to instability during training, requiring arduous hyperparameter searches to stabilize training and achieve high generalization and being very computationally expensive at both training and inference times. In this paper, we propose various modifications to MAML that not only stabilize the system, but also substantially improve the generalization performance, convergence speed and computational overhead of MAML, which we call MAML++.

arXiv commentPublished in ICLR 2019

References in corpus (14)

• Decoupled Weight Decay Regularization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Neural Architecture Search with Reinforcement Learning
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Data Augmentation Generative Adversarial Networks
• FractalNet: Ultra-Deep Neural Networks without Residuals
• On First-Order Meta-Learning Algorithms
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Learning to reinforcement learn
• How Does Batch Normalization Help Optimization?
• Using Fast Weights to Attend to the Recent Past
• Learning to Optimize
• Towards a Neural Statistician

Cited by in corpus (38)

• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Learning to Demodulate from Few Pilots via Offline and Online Meta-Learning
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• Generalized Inner Loop Meta-Learning
• Assume, Augment and Learn: Unsupervised Few-Shot Meta-Learning via Random Labels and Data Augmentation
• Meta-Graph: Few Shot Link Prediction via Meta Learning
• Learning How to Demodulate from Few Pilots via Meta-Learning
• Robust MAML: Prioritization task buffer with adaptive learning process for model-agnostic meta-learning
• Meta-Learning Symmetries by Reparameterization
• Deep Transformers with Latent Depth
• Meta-Learning Deep Energy-Based Memory Models
• Meta-Learning for Few-Shot Land Cover Classification
• Few-shot Action Recognition with Permutation-invariant Attention
• Towards Cross-Granularity Few-Shot Learning: Coarse-to-Fine Pseudo-Labeling with Visual-Semantic Meta-Embedding
• MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaption
• Few-shot Classification via Adaptive Attention
• One-Shot Object Localization Using Learnt Visual Cues via Siamese Networks
• Complementary Meta-Reinforcement Learning for Fault-Adaptive Control
• Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning
• Meta-Learning Initializations for Image Segmentation
• Formulating Camera-Adaptive Color Constancy as a Few-shot Meta-Learning Problem
• How Fine-Tuning Allows for Effective Meta-Learning
• Adaptive Task Sampling for Meta-Learning
• Rethinking Class Relations: Absolute-relative Supervised and Unsupervised Few-shot Learning
• Sign-MAML: Efficient Model-Agnostic Meta-Learning by SignSGD
• Physics-aware Spatiotemporal Modules with Auxiliary Tasks for Meta-Learning
• ReMP: Rectified Metric Propagation for Few-Shot Learning
• Improving Few-shot Learning by Spatially-aware Matching and CrossTransformer
• Few-shot Continual Learning: a Brain-inspired Approach
• VIABLE: Fast Adaptation via Backpropagating Learned Loss
• Population-Based Evolution Optimizes a Meta-Learning Objective
• Memory Efficient Meta-Learning with Large Images
• Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling
• Few-shot Learning via Dependency Maximization and Instance Discriminant Analysis
• Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?
• Fair Meta-Learning For Few-Shot Classification
• Meta-Learning to Improve Pre-Training
• Unsupervised Few-Shot Action Recognition via Action-Appearance Aligned Meta-Adaptation