Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark

Abstract

Meta and transfer learning are two successful families of approaches to few-shot learning. Despite highly related goals, state-of-the-art advances in each family are measured largely in isolation of each other. As a result of diverging evaluation norms, a direct or thorough comparison of different approaches is challenging. To bridge this gap, we perform a cross-family study of the best transfer and meta learners on both a large-scale meta-learning benchmark (Meta-Dataset, MD), and a transfer learning benchmark (Visual Task Adaptation Benchmark, VTAB). We find that, on average, large-scale transfer methods (Big Transfer, BiT) outperform competing approaches on MD, even when trained only on ImageNet. In contrast, meta-learning approaches struggle to compete on VTAB when trained and validated on MD. However, BiT is not without limitations, and pushing for scale does not improve performance on highly out-of-distribution MD tasks. In performing this study, we reveal a number of discrepancies in evaluation norms and study some of these in light of the performance gap. We hope that this work facilitates sharing of insights from each community, and accelerates progress on few-shot learning.

References in corpus (14)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Prototypical Networks for Few-shot Learning
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Deep Visual Domain Adaptation: A Survey
• What makes ImageNet good for transfer learning?
• Self-training with Noisy Student improves ImageNet classification
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation
• In Search of Lost Domain Generalization
• Learning from Noisy Labels with Distillation
• Optimized Generic Feature Learning for Few-shot Classification across Domains
• Feature Transformation Ensemble Model with Batch Spectral Regularization for Cross-Domain Few-Shot Classification
• Cross-Domain Few-Shot Learning with Meta Fine-Tuning
• Explanation-Guided Training for Cross-Domain Few-Shot Classification

Cited by in corpus (3)

• Exploring the Limits of Large Scale Pre-training
• A comprehensive and easy-to-use multi-domain multi-task medical imaging meta-dataset
• Memory Efficient Meta-Learning with Large Images