Discriminative Distillation to Reduce Class Confusion in Continual Learning

Abstract

Successful continual learning of new knowledge would enable intelligent systems to recognize more and more classes of objects. However, current intelligent systems often fail to correctly recognize previously learned classes of objects when updated to learn new classes. It is widely believed that such downgraded performance is solely due to the catastrophic forgetting of previously learned knowledge. In this study, we argue that the class confusion phenomena may also play a role in downgrading the classification performance during continual learning, i.e., the high similarity between new classes and any previously learned classes would also cause the classifier to make mistakes in recognizing these old classes, even if the knowledge of these old classes is not forgotten. To alleviate the class confusion issue, we propose a discriminative distillation strategy to help the classify well learn the discriminative features between confusing classes during continual learning. Experiments on multiple natural image classification tasks support that the proposed distillation strategy, when combined with existing methods, is effective in further improving continual learning.

arXiv admin note: text overlap with arXiv:2104.13614

References in corpus (5)

• Distilling the Knowledge in a Neural Network
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Compacting, Picking and Growing for Unforgetting Continual Learning
• Continual Unsupervised Representation Learning
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting