Semantic Drift Compensation for Class-Incremental Learning

Abstract

Class-incremental learning of deep networks sequentially increases the number of classes to be classified. During training, the network has only access to data of one task at a time, where each task contains several classes. In this setting, networks suffer from catastrophic forgetting which refers to the drastic drop in performance on previous tasks. The vast majority of methods have studied this scenario for classification networks, where for each new task the classification layer of the network must be augmented with additional weights to make room for the newly added classes. Embedding networks have the advantage that new classes can be naturally included into the network without adding new weights. Therefore, we study incremental learning for embedding networks. In addition, we propose a new method to estimate the drift, called semantic drift, of features and compensate for it without the need of any exemplars. We approximate the drift of previous tasks based on the drift that is experienced by current task data. We perform experiments on fine-grained datasets, CIFAR100 and ImageNet-Subset. We demonstrate that embedding networks suffer significantly less from catastrophic forgetting. We outperform existing methods which do not require exemplars and obtain competitive results compared to methods which store exemplars. Furthermore, we show that our proposed SDC when combined with existing methods to prevent forgetting consistently improves results.

Accepted at CVPR2020, Code available at \url{https://github.com/yulu0724/SDC-IL}

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Overcoming catastrophic forgetting in neural networks
• Prototypical Networks for Few-shot Learning
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• Three scenarios for continual learning
• Significance of Softmax-based Features in Comparison to Distance Metric Learning-based Features
• Metric Learning for Novelty and Anomaly Detection
• Memory Replay GANs: learning to generate images from new categories without forgetting
• Adapted Deep Embeddings: A Synthesis of Methods for $k$-Shot Inductive Transfer Learning
• End-to-End Incremental Learning

Cited by in corpus (5)

• A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
• RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning
• Co-Transport for Class-Incremental Learning
• Incremental Embedding Learning via Zero-Shot Translation
• Online Continual Learning in Image Classification: An Empirical Survey