Subspace Regularizers for Few-Shot Class Incremental Learning

Abstract

Few-shot class incremental learning -- the problem of updating a trained classifier to discriminate among an expanded set of classes with limited labeled data -- is a key challenge for machine learning systems deployed in non-stationary environments. Existing approaches to the problem rely on complex model architectures and training procedures that are difficult to tune and re-use. In this paper, we present an extremely simple approach that enables the use of ordinary logistic regression classifiers for few-shot incremental learning. The key to this approach is a new family of subspace regularization schemes that encourage weight vectors for new classes to lie close to the subspace spanned by the weights of existing classes. When combined with pretrained convolutional feature extractors, logistic regression models trained with subspace regularization outperform specialized, state-of-the-art approaches to few-shot incremental image classification by up to 22% on the miniImageNet dataset. Because of its simplicity, subspace regularization can be straightforwardly extended to incorporate additional background information about the new classes (including class names and descriptions specified in natural language); these further improve accuracy by up to 2%. Our results show that simple geometric regularization of class representations offers an effective tool for continual learning.

arXiv commentICLR 2022. Code is available through https://github.com/feyzaakyurek/subspace-reg

References in corpus (10)

• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Learning Transferable Visual Models From Natural Language Supervision
• Prototypical Networks for Few-shot Learning
• A continual learning survey: Defying forgetting in classification tasks
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Zero-Shot Learning Through Cross-Modal Transfer
• On Tiny Episodic Memories in Continual Learning
• Invariant Risk Minimization
• Clustered Multi-Task Learning: A Convex Formulation

Cited by in corpus (3)

• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning
• PKI: Prior Knowledge-Infused Neural Network for Few-Shot Class-Incremental Learning
• Continual Learning: Forget-free Winning Subnetworks for Video Representations