Information Theoretic Meta Learning with Gaussian Processes

Abstract

We formulate meta learning using information theoretic concepts; namely, mutual information and the information bottleneck. The idea is to learn a stochastic representation or encoding of the task description, given by a training set, that is highly informative about predicting the validation set. By making use of variational approximations to the mutual information, we derive a general and tractable framework for meta learning. This framework unifies existing gradient-based algorithms and also allows us to derive new algorithms. In particular, we develop a memory-based algorithm that uses Gaussian processes to obtain non-parametric encoding representations. We demonstrate our method on a few-shot regression problem and on four few-shot classification problems, obtaining competitive accuracy when compared to existing baselines.

arXiv comment15 pages, 2 figures

References in corpus (14)

• Auto-Encoding Variational Bayes
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• The information bottleneck method
• On First-Order Meta-Learning Algorithms
• EMNIST: an extension of MNIST to handwritten letters
• Understanding disentangling in $\beta$-VAE
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Compressing Neural Networks using the Variational Information Bottleneck
• Relevant sparse codes with variational information bottleneck
• TaskNorm: Rethinking Batch Normalization for Meta-Learning
• Adaptive Deep Kernel Learning
• Uncertainty in Multitask Transfer Learning
• Modular Meta-Learning with Shrinkage

Cited by in corpus (5)

• Bayesian Few-Shot Classification with One-vs-Each Pólya-Gamma Augmented Gaussian Processes
• Towards a population-informed approach to the definition of data-driven models for structural dynamics
• Non-Gaussian Gaussian Processes for Few-Shot Regression
• ST-MAML: A Stochastic-Task based Method for Task-Heterogeneous Meta-Learning
• Bayesian Meta-Learning Through Variational Gaussian Processes