Stateless Neural Meta-Learning using Second-Order Gradients

Abstract

Deep learning typically requires large data sets and much compute power for each new problem that is learned. Meta-learning can be used to learn a good prior that facilitates quick learning, thereby relaxing these requirements so that new tasks can be learned quicker; two popular approaches are MAML and the meta-learner LSTM. In this work, we compare the two and formally show that the meta-learner LSTM subsumes MAML. Combining this insight with recent empirical findings, we construct a new algorithm (dubbed TURTLE) which is simpler than the meta-learner LSTM yet more expressive than MAML. TURTLE outperforms both techniques at few-shot sine wave regression and image classification on miniImageNet and CUB without any additional hyperparameter tuning, at a computational cost that is comparable with second-order MAML. The key to TURTLE's success lies in the use of second-order gradients, which also significantly increases the performance of the meta-learner LSTM by 1-6% accuracy.

References in corpus (11)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• A Survey of Deep Meta-Learning
• Learning to learn by gradient descent by gradient descent
• Meta-Learning with Implicit Gradients
• Meta-Learning: A Survey
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace
• Understanding and correcting pathologies in the training of learned optimizers