Class-incremental Learning with Pre-allocated Fixed Classifiers

Abstract

In class-incremental learning, a learning agent faces a stream of data with the goal of learning new classes while not forgetting previous ones. Neural networks are known to suffer under this setting, as they forget previously acquired knowledge. To address this problem, effective methods exploit past data stored in an episodic memory while expanding the final classifier nodes to accommodate the new classes. In this work, we substitute the expanding classifier with a novel fixed classifier in which a number of pre-allocated output nodes are subject to the classification loss right from the beginning of the learning phase. Contrarily to the standard expanding classifier, this allows: (a) the output nodes of future unseen classes to firstly see negative samples since the beginning of learning together with the positive samples that incrementally arrive; (b) to learn features that do not change their geometric configuration as novel classes are incorporated in the learning model. Experiments with public datasets show that the proposed approach is as effective as the expanding classifier while exhibiting novel intriguing properties of the internal feature representation that are otherwise not-existent. Our ablation study on pre-allocating a large number of classes further validates the approach.

arXiv commentICPR 2021 (figure and typos fixed)

References in corpus (21)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Overcoming catastrophic forgetting in neural networks
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• NormFace: L2 Hypersphere Embedding for Face Verification
• Efficient Lifelong Learning with A-GEM
• Three scenarios for continual learning
• Progress & Compress: A scalable framework for continual learning
• Variational Continual Learning
• Mnemonics Training: Multi-Class Incremental Learning without Forgetting
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Lifelong Learning with Dynamically Expandable Networks
• Towards Robust Evaluations of Continual Learning
• Less-forgetting Learning in Deep Neural Networks
• Functional Regularisation for Continual Learning with Gaussian Processes
• Continual Unsupervised Representation Learning
• Memory Based Online Learning of Deep Representations from Video Streams
• Supermasks in Superposition
• Fix Your Features: Stationary and Maximally Discriminative Embeddings using Regular Polytope (Fixed Classifier) Networks
• Maximally Compact and Separated Features with Regular Polytope Networks
• Do We Need Fully Connected Output Layers in Convolutional Networks?

Cited by in corpus (3)

• Class-Incremental Continual Learning into the eXtended DER-verse
• Contrastive Supervised Distillation for Continual Representation Learning
• CoReS: Compatible Representations via Stationarity