Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis
arXiv:2109.14595
Abstract
We derive a novel information-theoretic analysis of the generalization property of meta-learning algorithms. Concretely, our analysis proposes a generic understanding of both the conventional learning-to-learn framework and the modern model-agnostic meta-learning (MAML) algorithms. Moreover, we provide a data-dependent generalization bound for a stochastic variant of MAML, which is non-vacuous for deep few-shot learning. As compared to previous bounds that depend on the square norm of gradients, empirical validations on both simulated data and a well-known few-shot benchmark show that our bound is orders of magnitude tighter in most situations.
References in corpus (16)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Prototypical Networks for Few-shot Learning
- Language Models are Few-Shot Learners
- Matching Networks for One Shot Learning
- Few-Shot Learning with Graph Neural Networks
- Improving Federated Learning Personalization via Model Agnostic Meta Learning
- Personalized Federated Learning: A Meta-Learning Approach
- Information-theoretic analysis of generalization capability of learning algorithms
- A PAC-Bayesian bound for Lifelong Learning
- The Benefit of Multitask Representation Learning
- Online Meta-Learning
- Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory
- Learning-to-Learn Stochastic Gradient Descent with Biased Regularization
- Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates
- Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning
- Reasoning About Generalization via Conditional Mutual Information