Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Nets
arXiv:2006.14606
Abstract
Gradient-based meta-learning (GBML) with deep neural nets (DNNs) has become a popular approach for few-shot learning. However, due to the non-convexity of DNNs and the bi-level optimization in GBML, the theoretical properties of GBML with DNNs remain largely unknown. In this paper, we first aim to answer the following question: Does GBML with DNNs have global convergence guarantees? We provide a positive answer to this question by proving that GBML with over-parameterized DNNs is guaranteed to converge to global optima at a linear rate. The second question we aim to address is: How does GBML achieve fast adaption to new tasks with prior experience on past tasks? To answer it, we theoretically show that GBML is equivalent to a functional gradient descent operation that explicitly propagates experience from the past tasks to new ones, and then we prove a generalization error bound of GBML with over-parameterized DNNs.
arXiv commentUnder review. Code available at https://github.com/AI-secure/Meta-Neural-Kernel
References in corpus (26)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Domain Generalization via Invariant Feature Representation
- On First-Order Meta-Learning Algorithms
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- Meta-Learning with Implicit Gradients
- Meta-Learning: A Survey
- Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
- Optimization for deep learning: theory and algorithms
- Gradient descent aligns the layers of deep linear networks
- The Benefit of Multitask Representation Learning
- Online Meta-Learning
- Generalized Inner Loop Meta-Learning
- Generalizing to unseen domains via distribution matching
- Enhanced Convolutional Neural Tangent Kernels
- On the loss landscape of a class of deep neural networks with no bad local valleys
- Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
- Understanding the Loss Surface of Neural Networks for Binary Classification
- Adding One Neuron Can Eliminate All Bad Local Minima
- Elimination of All Bad Local Minima in Deep Learning
- On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
- Biased Stochastic First-Order Methods for Conditional Stochastic Optimization and Applications in Meta Learning
- Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinity
- On the Global Optimality of Model-Agnostic Meta-Learning
- A Sample Complexity Separation between Non-Convex and Convex Meta-Learning