Are Few-Shot Learning Benchmarks too Simple ? Solving them without Task Supervision at Test-Time

Abstract

We show that several popular few-shot learning benchmarks can be solved with varying degrees of success without using support set Labels at Test-time (LT). To this end, we introduce a new baseline called Centroid Networks, a modification of Prototypical Networks in which the support set labels are hidden from the method at test-time and have to be recovered through clustering. A benchmark that can be solved perfectly without LT does not require proper task adaptation and is therefore inadequate for evaluating few-shot methods. In practice, most benchmarks cannot be solved perfectly without LT, but running our baseline on any new combinations of architectures and datasets gives insights on the baseline performance to be expected from leveraging a good representation, before any adaptation to the test-time labels.

References in corpus (15)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Zero-Shot Learning Through Cross-Modal Transfer
• Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning
• Fast Computation of Wasserstein Barycenters
• Learning to Remember Rare Events
• SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning
• A Closer Look at Few-shot Classification
• Unsupervised Learning via Meta-Learning
• Towards a Neural Statistician
• Cross Attention Network for Few-shot Classification
• Learning to cluster in order to transfer across domains and tasks
• Transductive Episodic-Wise Adaptive Metric for Few-Shot Learning
• Negative Margin Matters: Understanding Margin in Few-shot Classification
• Learning Probability Measures with respect to Optimal Transport Metrics

Cited by in corpus (5)

• Leveraging the Feature Distribution in Transfer-based Few-Shot Learning
• Learning to Discover Novel Visual Categories via Deep Transfer Clustering
• Deep Amortized Clustering
• Zero-Shot Learning from scratch (ZFS): leveraging local compositional representations
• Unsupervised Transfer Learning with Self-Supervised Remedy