Sign-MAML: Efficient Model-Agnostic Meta-Learning by SignSGD

Abstract

We propose a new computationally-efficient first-order algorithm for Model-Agnostic Meta-Learning (MAML). The key enabling technique is to interpret MAML as a bilevel optimization (BLO) problem and leverage the sign-based SGD(signSGD) as a lower-level optimizer of BLO. We show that MAML, through the lens of signSGD-oriented BLO, naturally yields an alternating optimization scheme that just requires first-order gradients of a learned meta-model. We term the resulting MAML algorithm Sign-MAML. Compared to the conventional first-order MAML (FO-MAML) algorithm, Sign-MAML is theoretically-grounded as it does not impose any assumption on the absence of second-order derivatives during meta training. In practice, we show that Sign-MAML outperforms FO-MAML in various few-shot image classification tasks, and compared to MAML, it achieves a much more graceful tradeoff between classification accuracy and computation efficiency.

References in corpus (19)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Neural Architecture Search with Reinforcement Learning
• Towards Deep Learning Models Resistant to Adversarial Attacks
• On First-Order Meta-Learning Algorithms
• Meta-Learning with Implicit Gradients
• TADAM: Task dependent adaptive metric for improved few-shot learning
• Meta-Learning: A Survey
• Fast Context Adaptation via Meta-Learning
• On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• How to train your MAML
• Approximation Methods for Bilevel Programming
• A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic
• ES-MAML: Simple Hessian-Free Meta Learning
• Bilevel Optimization: Convergence Analysis and Enhanced Design
• Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameters
• When does MAML Work the Best? An Empirical Study on Model-Agnostic Meta-Learning in NLP Applications
• Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks