Theoretical bounds on estimation error for meta-learning
arXiv:2010.07140
Abstract
Machine learning models have traditionally been developed under the assumption that the training and test distributions match exactly. However, recent success in few-shot learning and related problems are encouraging signs that these models can be adapted to more realistic settings where train and test distributions differ. Unfortunately, there is severely limited theoretical support for these algorithms and little is known about the difficulty of these problems. In this work, we provide novel information-theoretic lower-bounds on minimax rates of convergence for algorithms that are trained on data from multiple sources and tested on novel data. Our bounds depend intuitively on the information shared between sources of data, and characterize the difficulty of learning in this setting for arbitrary algorithms. We demonstrate these bounds on a hierarchical Bayesian model of meta-learning, computing both upper and lower bounds on parameter estimation via maximum-a-posteriori inference.
arXiv comment12 pages in main paper,22 pages in appendix,4 figures total
References in corpus (13)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Prototypical Networks for Few-shot Learning
- A Theoretical Analysis of Contrastive Unsupervised Representation Learning
- Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
- A PAC-Bayesian bound for Lifelong Learning
- Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory
- Meta-Learning Update Rules for Unsupervised Representation Learning
- Few-Shot Learning via Learning the Representation, Provably
- Learning-to-Learn Stochastic Gradient Descent with Biased Regularization
- Provable Guarantees for Gradient-Based Meta-Learning
- Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions
- On the Value of Target Data in Transfer Learning
- A Theoretical Analysis of the Number of Shots in Few-Shot Learning