Meta-Learning with Adaptive Hyperparameters

Abstract

Despite its popularity, several recent works question the effectiveness of MAML when test tasks are different from training tasks, thus suggesting various task-conditioned methodology to improve the initialization. Instead of searching for better task-aware initialization, we focus on a complementary factor in MAML framework, inner-loop optimization (or fast adaptation). Consequently, we propose a new weight update rule that greatly enhances the fast adaptation process. Specifically, we introduce a small meta-network that can adaptively generate per-step hyperparameters: learning rate and weight decay coefficients. The experimental results validate that the Adaptive Learning of hyperparameters for Fast Adaptation (ALFA) is the equally important ingredient that was often neglected in the recent few-shot learning approaches. Surprisingly, fast adaptation from random initialization with ALFA can already outperform MAML.

NeurIPS 2020. Code at https://github.com/baiksung/alfa. Typo fix in the updated version

References in corpus (7)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• A Simple Neural Attentive Meta-Learner
• Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning
• TADAM: Task dependent adaptive metric for improved few-shot learning
• Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?

Cited by in corpus (2)

• Signal Transformer: Complex-valued Attention and Meta-Learning for Signal Recognition
• ST-MAML: A Stochastic-Task based Method for Task-Heterogeneous Meta-Learning