Maintaining Discrimination and Fairness in Class Incremental Learning

Abstract

Deep neural networks (DNNs) have been applied in class incremental learning, which aims to solve common real-world problems of learning new classes continually. One drawback of standard DNNs is that they are prone to catastrophic forgetting. Knowledge distillation (KD) is a commonly used technique to alleviate this problem. In this paper, we demonstrate it can indeed help the model to output more discriminative results within old classes. However, it cannot alleviate the problem that the model tends to classify objects into new classes, causing the positive effect of KD to be hidden and limited. We observed that an important factor causing catastrophic forgetting is that the weights in the last fully connected (FC) layer are highly biased in class incremental learning. In this paper, we propose a simple and effective solution motivated by the aforementioned observations to address catastrophic forgetting. Firstly, we utilize KD to maintain the discrimination within old classes. Then, to further maintain the fairness between old classes and new classes, we propose Weight Aligning (WA) that corrects the biased weights in the FC layer after normal training process. Unlike previous work, WA does not require any extra parameters or a validation set in advance, as it utilizes the information provided by the biased weights themselves. The proposed method is evaluated on ImageNet-1000, ImageNet-100, and CIFAR-100 under various settings. Experimental results show that the proposed method can effectively alleviate catastrophic forgetting and significantly outperform state-of-the-art methods.

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• Deep Residual Learning for Image Recognition
• A systematic study of the class imbalance problem in convolutional neural networks
• Densely Connected Convolutional Networks
• Continual Learning Through Synaptic Intelligence
• Three scenarios for continual learning
• Variational Continual Learning
• Continual Learning with Deep Generative Replay
• One-shot Face Recognition by Promoting Underrepresented Classes
• M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning
• Class-incremental Learning via Deep Model Consolidation
• Label Mapping Neural Networks with Response Consolidation for Class Incremental Learning

Cited by in corpus (5)

• DER: Dynamically Expandable Representation for Class Incremental Learning
• Leveraging Old Knowledge to Continually Learn New Classes in Medical Images
• ADER: Adaptively Distilled Exemplar Replay Towards Continual Learning for Session-based Recommendation
• Unsupervised Continual Learning Via Pseudo Labels
• ClaRe: Practical Class Incremental Learning By Remembering Previous Class Representations