Embedding Adaptation is Still Needed for Few-Shot Learning

Abstract

Constructing new and more challenging tasksets is a fruitful methodology to analyse and understand few-shot classification methods. Unfortunately, existing approaches to building those tasksets are somewhat unsatisfactory: they either assume train and test task distributions to be identical -- which leads to overly optimistic evaluations -- or take a "worst-case" philosophy -- which typically requires additional human labor such as obtaining semantic class relationships. We propose ATG, a principled clustering method to defining train and test tasksets without additional human knowledge. ATG models train and test task distributions while requiring them to share a predefined amount of information. We empirically demonstrate the effectiveness of ATG in generating tasksets that are easier, in-between, or harder than existing benchmarks, including those that rely on semantic information. Finally, we leverage our generated tasksets to shed a new light on few-shot classification: gradient-based methods -- previously believed to underperform -- can outperform metric-based ones when transfer is most challenging.

In submission

References in corpus (18)

• Scikit-learn: Machine Learning in Python
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Prototypical Networks for Few-shot Learning
• Fine-Grained Visual Classification of Aircraft
• A Simple Neural Attentive Meta-Learner
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Multi-Task Learning as Multi-Objective Optimization
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Meta-Learning with Implicit Gradients
• Meta-Learning: A Survey
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Online Meta-Learning
• Alpha MAML: Adaptive Model-Agnostic Meta-Learning
• Understanding and Improving Information Transfer in Multi-Task Learning
• Meta-learning of Sequential Strategies
• Meta-Learning Symmetries by Reparameterization
• Measuring and Harnessing Transference in Multi-Task Learning
• The TCGA Meta-Dataset Clinical Benchmark

Cited by in corpus (2)

• How to Train Your MAML to Excel in Few-Shot Classification
• Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution