Function Contrastive Learning of Transferable Meta-Representations

Abstract

Meta-learning algorithms adapt quickly to new tasks that are drawn from the same task distribution as the training tasks. The mechanism leading to fast adaptation is the conditioning of a downstream predictive model on the inferred representation of the task's underlying data generative process, or \emph{function}. This \emph{meta-representation}, which is computed from a few observed examples of the underlying function, is learned jointly with the predictive model. In this work, we study the implications of this joint training on the transferability of the meta-representations. Our goal is to learn meta-representations that are robust to noise in the data and facilitate solving a wide range of downstream tasks that share the same underlying functions. To this end, we propose a decoupled encoder-decoder approach to supervised meta-learning, where the encoder is trained with a contrastive objective to find a good representation of the underlying function. In particular, our training scheme is driven by the self-supervision signal indicating whether two sets of examples stem from the same function. Our experiments on a number of synthetic and real-world datasets show that the representations we obtain outperform strong baselines in terms of downstream performance and noise robustness, even when these baselines are trained in an end-to-end manner.

arXiv commentICML 2021

References in corpus (18)

• A Simple Framework for Contrastive Learning of Visual Representations
• Prototypical Networks for Few-shot Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Learning deep representations by mutual information estimation and maximization
• Momentum Contrast for Unsupervised Visual Representation Learning
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• On Mutual Information Maximization for Representation Learning
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Infinite Mixture Prototypes for Few-Shot Learning
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• Contrastive Learning of Structured World Models
• CrossTransformers: spatially-aware few-shot transfer
• Conditional Neural Processes
• Attentive Neural Processes
• Noise Contrastive Meta-Learning for Conditional Density Estimation using Kernel Mean Embeddings
• MetaFun: Meta-Learning with Iterative Functional Updates