Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stability

Abstract

We are motivated by the problem of providing strong generalization guarantees in the context of meta-learning. Existing generalization bounds are either challenging to evaluate or provide vacuous guarantees in even relatively simple settings. We derive a probably approximately correct (PAC) bound for gradient-based meta-learning using two different generalization frameworks in order to deal with the qualitatively different challenges of generalization at the "base" and "meta" levels. We employ bounds for uniformly stable algorithms at the base level and bounds from the PAC-Bayes framework at the meta level. The result of this approach is a novel PAC bound that is tighter when the base learner adapts quickly, which is precisely the goal of meta-learning. We show that our bound provides a tighter guarantee than other bounds on a toy non-convex problem on the unit sphere and a text-based classification example. We also present a practical regularization scheme motivated by the bound in settings where the bound is loose and demonstrate improved performance over baseline techniques.

References in corpus (17)

• Scikit-learn: Machine Learning in Python
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Theoretical Models of Learning to Learn
• On First-Order Meta-Learning Algorithms
• Certifying Some Distributional Robustness with Principled Adversarial Training
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• A PAC-Bayesian bound for Lifelong Learning
• A Note on the PAC Bayesian Theorem
• Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory
• A PAC-Bayesian Tutorial with A Dropout Bound
• A PAC-Bayesian Analysis of Randomized Learning with Application to Stochastic Gradient Descent
• Tighter risk certificates for neural networks
• A Strongly Quasiconvex PAC-Bayesian Bound
• PAC-Bayes with Backprop
• PAC-Bayes bounds for stable algorithms with instance-dependent priors
• Stability revisited: new generalisation bounds for the Leave-one-Out