SupportNet: solving catastrophic forgetting in class incremental learning with support data

Abstract

A plain well-trained deep learning model often does not have the ability to learn new knowledge without forgetting the previously learned knowledge, which is known as catastrophic forgetting. Here we propose a novel method, SupportNet, to efficiently and effectively solve the catastrophic forgetting problem in the class incremental learning scenario. SupportNet combines the strength of deep learning and support vector machine (SVM), where SVM is used to identify the support data from the old data, which are fed to the deep learning model together with the new data for further training so that the model can review the essential information of the old data when learning the new information. Two powerful consolidation regularizers are applied to stabilize the learned representation and ensure the robustness of the learned model. We validate our method with comprehensive experiments on various tasks, which show that SupportNet drastically outperforms the state-of-the-art incremental learning methods and even reaches similar performance as the deep learning model trained from scratch on both old and new data. Our program is accessible at: https://github.com/lykaust15/SupportNet

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Overcoming catastrophic forgetting in neural networks
• Going Deeper with Convolutions
• Gradient Episodic Memory for Continual Learning
• Aggregated Residual Transformations for Deep Neural Networks
• Variational Continual Learning
• Continual Learning with Deep Generative Replay
• Less-forgetting Learning in Deep Neural Networks
• On the Decision Boundary of Deep Neural Networks

Cited by in corpus (9)

• Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
• M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning
• On the Decision Boundary of Deep Neural Networks
• Deep learning in bioinformatics: introduction, application, and perspective in big data era
• Overcoming Catastrophic Forgetting by Generative Regularization
• Frosting Weights for Better Continual Training
• Recent Advances in Large Margin Learning
• Do Not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting