CrossTransformers: spatially-aware few-shot transfer

Abstract

Given new tasks with very little data$-$such as new classes in a classification problem or a domain shift in the input$-$performance of modern vision systems degrades remarkably quickly. In this work, we illustrate how the neural network representations which underpin modern vision systems are subject to supervision collapse, whereby they lose any information that is not necessary for performing the training task, including information that may be necessary for transfer to new tasks or domains. We then propose two methods to mitigate this problem. First, we employ self-supervised learning to encourage general-purpose features that transfer better. Second, we propose a novel Transformer based neural network architecture called CrossTransformers, which can take a small number of labeled images and an unlabeled query, find coarse spatial correspondence between the query and the labeled images, and then infer class membership by computing distances between spatially-corresponding features. The result is a classifier that is more robust to task and domain shift, which we demonstrate via state-of-the-art performance on Meta-Dataset, a recent dataset for evaluating transfer from ImageNet to many other vision datasets.

arXiv commentPublished at NeurIPS 2020. Code/checkpoints: https://github.com/google-research/meta-dataset

References in corpus (12)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• WaveNet: A Generative Model for Raw Audio
• Self-Attention Generative Adversarial Networks
• Theoretical Models of Learning to Learn
• Fine-Grained Visual Classification of Aircraft
• A Simple Neural Attentive Meta-Learner
• Learning multiple visual domains with residual adapters
• Contrastive Multiview Coding
• Few-Shot Learning with Graph Neural Networks
• Learning to Remember Rare Events
• Optimized Generic Feature Learning for Few-shot Classification across Domains

Cited by in corpus (10)

• Transformers in Vision: A Survey
• A Closer Look at Few-Shot 3D Point Cloud Classification
• MCML: A Novel Memory-based Contrastive Meta-Learning Method for Few Shot Slot Tagging
• Function Contrastive Learning of Transferable Meta-Representations
• KNN Transformer with Pyramid Prompts for Few-Shot Learning
• One to Transfer All: A Universal Transfer Framework for Vision Foundation Model with Few Data
• Memory Efficient Meta-Learning with Large Images
• Hybrid BYOL-ViT: Efficient approach to deal with small datasets
• The Best of Both Worlds: a Framework for Combining Degradation Prediction with High Performance Super-Resolution Networks
• A${}^2$M${}^2$-Net: Adaptively Aligned Multi-Scale Moment for Few-Shot Action Recognition