Towards Understanding Generalization in Gradient-Based Meta-Learning

Abstract

In this work we study generalization of neural networks in gradient-based meta-learning by analyzing various properties of the objective landscapes. We experimentally demonstrate that as meta-training progresses, the meta-test solutions, obtained after adapting the meta-train solution of the model, to new tasks via few steps of gradient-based fine-tuning, become flatter, lower in loss, and further away from the meta-train solution. We also show that those meta-test solutions become flatter even as generalization starts to degrade, thus providing an experimental evidence against the correlation between generalization and flat minima in the paradigm of gradient-based meta-leaning. Furthermore, we provide empirical evidence that generalization to new tasks is correlated with the coherence between their adaptation trajectories in parameter space, measured by the average cosine similarity between task-specific trajectory directions, starting from a same meta-train solution. We also show that coherence of meta-test gradients, measured by the average inner product between the task-specific gradient vectors evaluated at meta-train solution, is also correlated with generalization. Based on these observations, we propose a novel regularizer for MAML and provide experimental evidence for its effectiveness.

References in corpus (7)

• Prototypical Networks for Few-shot Learning
• Matching Networks for One Shot Learning
• Visualizing the Loss Landscape of Neural Nets
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Three Factors Influencing Minima in SGD
• Essentially No Barriers in Neural Network Energy Landscape
• On the saddle point problem for non-convex optimization

Cited by in corpus (9)

• Meta-Learning without Memorization
• Meta-Learning Requires Meta-Augmentation
• When MAML Can Adapt Fast and How to Assist When It Cannot
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• Meta-Learning with Fewer Tasks through Task Interpolation
• Generalization of Model-Agnostic Meta-Learning Algorithms: Recurring and Unseen Tasks
• Efficient hyperparameter optimization by way of PAC-Bayes bound minimization
• Local Nonparametric Meta-Learning
• TAG: Task-based Accumulated Gradients for Lifelong learning