Less-forgetting Learning in Deep Neural Networks

Abstract

A catastrophic forgetting problem makes deep neural networks forget the previously learned information, when learning data collected in new environments, such as by different sensors or in different light conditions. This paper presents a new method for alleviating the catastrophic forgetting problem. Unlike previous research, our method does not use any information from the source domain. Surprisingly, our method is very effective to forget less of the information in the source domain, and we show the effectiveness of our method using several experiments. Furthermore, we observed that the forgetting problem occurs between mini-batches when performing general training processes using stochastic gradient descent methods, and this problem is one of the factors that degrades generalization performance of the network. We also try to solve this problem using the proposed method. Finally, we show our less-forgetting learning method is also helpful to improve the performance of deep neural networks in terms of recognition rates.

5 pages, 4 figures

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors
• Going Deeper with Convolutions
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks

Cited by in corpus (35)

• A continual learning survey: Defying forgetting in classification tasks
• Overcoming catastrophic forgetting with hard attention to the task
• Towards Robust Evaluations of Continual Learning
• Incremental Classifier Learning with Generative Adversarial Networks
• Lifelong Teacher-Student Network Learning
• Inexact-ADMM Based Federated Meta-Learning for Fast and Continual Edge Learning
• CLeaR: An Adaptive Continual Learning Framework for Regression Tasks
• A Strategy for an Uncompromising Incremental Learner
• Class-incremental Learning with Pre-allocated Fixed Classifiers
• Continual Learning with Adaptive Weights (CLAW)
• Toward Continual Learning for Conversational Agents
• Uncertainty-guided Continual Learning with Bayesian Neural Networks
• SupportNet: solving catastrophic forgetting in class incremental learning with support data
• Two-Level Residual Distillation based Triple Network for Incremental Object Detection
• Batch-level Experience Replay with Review for Continual Learning
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting
• Incremental Learning In Online Scenario
• Learning latent representations across multiple data domains using Lifelong VAEGAN
• Continual Representation Learning for Biometric Identification
• On the importance of cross-task features for class-incremental learning
• Differentially Private Continual Learning
• How to Evaluate the Next System: Automatic Dialogue Evaluation from the Perspective of Continual Learning
• Layerwise Optimization by Gradient Decomposition for Continual Learning
• Lifelong Learning with Sketched Structural Regularization
• Reviewing continual learning from the perspective of human-level intelligence
• Meta-Learned Attribute Self-Gating for Continual Generalized Zero-Shot Learning
• CAM-GAN: Continual Adaptation Modules for Generative Adversarial Networks
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Large Scale Incremental Learning
• Incremental Concept Learning via Online Generative Memory Recall
• Collaborative Method for Incremental Learning on Classification and Generation
• Lifelong Mixture of Variational Autoencoders
• iLGaCo: Incremental Learning of Gait Covariate Factors
• SeNA-CNN: Overcoming Catastrophic Forgetting in Convolutional Neural Networks by Selective Network Augmentation
• Center Loss Regularization for Continual Learning