Defining Benchmarks for Continual Few-Shot Learning

Abstract

Both few-shot and continual learning have seen substantial progress in the last years due to the introduction of proper benchmarks. That being said, the field has still to frame a suite of benchmarks for the highly desirable setting of continual few-shot learning, where the learner is presented a number of few-shot tasks, one after the other, and then asked to perform well on a validation set stemming from all previously seen tasks. Continual few-shot learning has a small computational footprint and is thus an excellent setting for efficient investigation and experimentation. In this paper we first define a theoretical framework for continual few-shot learning, taking into account recent literature, then we propose a range of flexible benchmarks that unify the evaluation criteria and allows exploring the problem from multiple perspectives. As part of the benchmark, we introduce a compact variant of ImageNet, called SlimageNet64, which retains all original 1000 classes but only contains 200 instances of each one (a total of 200K data-points) downscaled to 64 x 64 pixels. We provide baselines for the proposed benchmarks using a number of popular few-shot learning algorithms, as a result, exposing previously unknown strengths and weaknesses of those algorithms in continual and data-limited settings.

References in corpus (15)

• Overcoming catastrophic forgetting in neural networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Prototypical Networks for Few-shot Learning
• Continual Learning Through Synaptic Intelligence
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Data Augmentation Generative Adversarial Networks
• Meta-Learning for Semi-Supervised Few-Shot Classification
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• State Representation Learning for Control: An Overview
• Measuring Catastrophic Forgetting in Neural Networks
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Toward Continual Learning for Conversational Agents
• Meta Continual Learning
• Meta-Learning Representations for Continual Learning

Cited by in corpus (11)

• Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels
• Self-Supervised Relational Reasoning for Representation Learning
• Wandering Within a World: Online Contextualized Few-Shot Learning
• Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning
• Continuous Coordination As a Realistic Scenario for Lifelong Learning
• Few-Shot Unsupervised Continual Learning through Meta-Examples
• Few-shot Continual Learning: a Brain-inspired Approach
• Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning
• Memory Efficient Meta-Learning with Large Images
• Variable-Shot Adaptation for Online Meta-Learning
• How Sensitive are Meta-Learners to Dataset Imbalance?