Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution

Abstract

We categorize meta-learning evaluation into two settings: $\text{in-distribution}$ [ID], in which the train and test tasks are sampled $\text{iid}$ from the same underlying task distribution, and $\text{out-of-distribution}$ [OOD], in which they are not. While most meta-learning theory and some FSL applications follow the ID setting, we identify that most existing few-shot classification benchmarks instead reflect OOD evaluation, as they use disjoint sets of train (base) and test (novel) classes for task generation. This discrepancy is problematic because -- as we show on numerous benchmarks -- meta-learning methods that perform better on existing OOD datasets may perform significantly worse in the ID setting. In addition, in the OOD setting, even though current FSL benchmarks seem befitting, our study highlights concerns in 1) reliably performing model selection for a given meta-learning method, and 2) consistently comparing the performance of different methods. To address these concerns, we provide suggestions on how to construct FSL benchmarks to allow for ID evaluation as well as more reliable OOD evaluation. Our work aims to inform the meta-learning community about the importance and distinction of ID vs. OOD evaluation, as well as the subtleties of OOD evaluation with current benchmarks.

References in corpus (14)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Domain Generalization via Invariant Feature Representation
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• LEAF: A Benchmark for Federated Settings
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Meta-Learning with Implicit Gradients
• In Search of Lost Domain Generalization
• Few-Shot Learning via Learning the Representation, Provably
• How Important is the Train-Validation Split in Meta-Learning?
• Is Support Set Diversity Necessary for Meta-Learning?
• Embedding Adaptation is Still Needed for Few-Shot Learning
• Generalization of Model-Agnostic Meta-Learning Algorithms: Recurring and Unseen Tasks