On Episodes, Prototypical Networks, and Few-shot Learning
arXiv:2012.09831
Abstract
Episodic learning is a popular practice among researchers and practitioners interested in few-shot learning. It consists of organising training in a series of learning problems (or episodes), each divided into a small training and validation subset to mimic the circumstances encountered during evaluation. But is this always necessary? In this paper, we investigate the usefulness of episodic learning in methods which use nonparametric approaches, such as nearest neighbours, at the level of the episode. For these methods, we not only show how the constraints imposed by episodic learning are not necessary, but that they in fact lead to a data-inefficient way of exploiting training batches. We conduct a wide range of ablative experiments with Matching and Prototypical Networks, two of the most popular methods that use nonparametric approaches at the level of the episode. Their "non-episodic" counterparts are considerably simpler, have less hyperparameters, and improve their performance in multiple few-shot classification datasets.
18 pages. To appear at NeurIPS 2021. A preliminary version of this work appeared as an oral presentation at the NeurIPS 2020 meta-learning workshop
References in corpus (10)
- A Simple Framework for Contrastive Learning of Visual Representations
- On Calibration of Modern Neural Networks
- SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning
- Supervised Contrastive Learning
- Generalized Inner Loop Meta-Learning
- Infinite Mixture Prototypes for Few-Shot Learning
- TapNet: Neural Network Augmented with Task-Adaptive Projection for Few-Shot Learning
- Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples
- Analyzing and Improving Representations with the Soft Nearest Neighbor Loss
- Torchmeta: A Meta-Learning library for PyTorch