Self-Supervised Learning Aided Class-Incremental Lifelong Learning

Abstract

Lifelong or continual learning remains to be a challenge for artificial neural network, as it is required to be both stable for preservation of old knowledge and plastic for acquisition of new knowledge. It is common to see previous experience get overwritten, which leads to the well-known issue of catastrophic forgetting, especially in the scenario of class-incremental learning (Class-IL). Recently, many lifelong learning methods have been proposed to avoid catastrophic forgetting. However, models which learn without replay of the input data, would encounter another problem which has been ignored, and we refer to it as prior information loss (PIL). In training procedure of Class-IL, as the model has no knowledge about following tasks, it would only extract features necessary for tasks learned so far, whose information is insufficient for joint classification. In this paper, our empirical results on several image datasets show that PIL limits the performance of current state-of-the-art method for Class-IL, the orthogonal weights modification (OWM) algorithm. Furthermore, we propose to combine self-supervised learning, which can provide effective representations without requiring labels, with Class-IL to partly get around this problem. Experiments show superiority of proposed method to OWM, as well as other strong baselines.

References in corpus (15)

• Overcoming catastrophic forgetting in neural networks
• Representation Learning with Contrastive Predictive Coding
• Learning deep representations by mutual information estimation and maximization
• Continual Learning Through Synaptic Intelligence
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Three scenarios for continual learning
• Variational Continual Learning
• Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
• Overcoming catastrophic forgetting with hard attention to the task
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Lifelong Learning with Dynamically Expandable Networks
• Deep Generative Dual Memory Network for Continual Learning
• Continual Unsupervised Representation Learning
• Memory Replay GANs: learning to generate images from new categories without forgetting

Cited by in corpus (2)

• Learning Representations for New Sound Classes With Continual Self-Supervised Learning
• Energy-Based Models for Continual Learning