Overcoming Catastrophic Forgetting by Incremental Moment Matching

Abstract

Catastrophic forgetting is a problem of neural networks that loses the information of the first task after training the second task. Here, we propose a method, i.e. incremental moment matching (IMM), to resolve this problem. IMM incrementally matches the moment of the posterior distribution of the neural network which is trained on the first and the second task, respectively. To make the search space of posterior parameter smooth, the IMM procedure is complemented by various transfer learning techniques including weight transfer, L2-norm of the old and the new parameter, and a variant of dropout with the old parameter. We analyze our approach on a variety of datasets including the MNIST, CIFAR-10, Caltech-UCSD-Birds, and Lifelog datasets. The experimental results show that IMM achieves state-of-the-art performance by balancing the information between an old and a new network.

arXiv commentAccepted for NIPS 2017

Cited by in corpus (103)

• A continual learning survey: Defying forgetting in classification tasks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• On Tiny Episodic Memories in Continual Learning
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Towards Robust Evaluations of Continual Learning
• Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data
• Compacting, Picking and Growing for Unforgetting Continual Learning
• Variational Federated Multi-Task Learning
• LAMOL: LAnguage MOdeling for Lifelong Language Learning
• Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
• Meta-Learning with Warped Gradient Descent
• Overcoming Long-term Catastrophic Forgetting through Adversarial Neural Pruning and Synaptic Consolidation
• Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks
• Cognitively-Inspired Model for Incremental Learning Using a Few Examples
• Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems
• Online Coreset Selection for Rehearsal-based Continual Learning
• Orthogonal Gradient Descent for Continual Learning
• Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting
• Controlled Forgetting: Targeted Stimulation and Dopaminergic Plasticity Modulation for Unsupervised Lifelong Learning in Spiking Neural Networks
• Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights
• An Investigation of Replay-based Approaches for Continual Learning
• Toward Understanding Catastrophic Forgetting in Continual Learning
• Federated Continual Learning with Weighted Inter-client Transfer
• Class-incremental Learning via Deep Model Consolidation
• Improved Schemes for Episodic Memory-based Lifelong Learning
• Uncertainty-guided Continual Learning with Bayesian Neural Networks
• Meta-Learning Representations for Continual Learning
• Continual Learning via Inter-Task Synaptic Mapping
• Lifelong Object Detection
• A Review of Single-Source Deep Unsupervised Visual Domain Adaptation
• Parameter-Efficient Transfer Learning with Diff Pruning
• Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
• Catastrophic forgetting: still a problem for DNNs
• Rainbow Memory: Continual Learning with a Memory of Diverse Samples
• Progressive Memory Banks for Incremental Domain Adaptation
• Beneficial Perturbation Network for designing general adaptive artificial intelligence systems
• Scalable and Order-robust Continual Learning with Additive Parameter Decomposition
• Generalized Variational Continual Learning
• Carousel Memory: Rethinking the Design of Episodic Memory for Continual Learning
• GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning
• Few-Shot Self Reminder to Overcome Catastrophic Forgetting
• Linear Mode Connectivity in Multitask and Continual Learning
• Overcoming catastrophic forgetting problem by weight consolidation and long-term memory
• Overcoming Catastrophic Interference by Conceptors
• Variational Prototype Replays for Continual Learning
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting
• Training Networks in Null Space of Feature Covariance for Continual Learning
• Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning
• Online Deep Learning based on Auto-Encoder
• Learning Adaptive Embedding Considering Incremental Class
• Continual Learning with Knowledge Transfer for Sentiment Classification
• ARCADe: A Rapid Continual Anomaly Detector
• Dissecting Catastrophic Forgetting in Continual Learning by Deep Visualization
• Two geometric input transformation methods for fast online reinforcement learning with neural nets
• Evaluating Online Continual Learning with CALM
• Attribution Preservation in Network Compression for Reliable Network Interpretation
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain
• Towards Generalized and Incremental Few-Shot Object Detection
• Bayesian Optimized Continual Learning with Attention Mechanism
• Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference
• Towards a General Model of Knowledge for Facial Analysis by Multi-Source Transfer Learning
• SpikeDyn: A Framework for Energy-Efficient Spiking Neural Networks with Continual and Unsupervised Learning Capabilities in Dynamic Environments
• Continual Learning Using Bayesian Neural Networks
• Posterior Meta-Replay for Continual Learning
• Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition
• Continual Learning with Fully Probabilistic Models
• Continual Learning Using Multi-view Task Conditional Neural Networks
• Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control
• Attention-Based Structural-Plasticity
• Single-Net Continual Learning with Progressive Segmented Training (PST)
• Optimizing Reusable Knowledge for Continual Learning via Metalearning
• Frosting Weights for Better Continual Training
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Memory Efficient Class-Incremental Learning for Image Classification
• Reviewing continual learning from the perspective of human-level intelligence
• Continual Learning with Extended Kronecker-factored Approximate Curvature
• Co-Transport for Class-Incremental Learning
• Learning Continually from Low-shot Data Stream
• Learning to Remember from a Multi-Task Teacher
• Representation Consolidation for Training Expert Students
• Incremental Concept Learning via Online Generative Memory Recall
• Adaptive Compression-based Lifelong Learning
• S2OSC: A Holistic Semi-Supervised Approach for Open Set Classification
• Unsupervised Domain Expansion from Multiple Sources
• Beneficial perturbation network for continual learning
• Efficient Multi-Domain Network Learning by Covariance Normalization
• A Biologically Plausible Audio-Visual Integration Model for Continual Learning
• Online Continual Learning in Image Classification: An Empirical Survey
• Class Incremental Online Streaming Learning
• Continual Learning of Generative Models with Limited Data: From Wasserstein-1 Barycenter to Adaptive Coalescence
• Do Not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting
• Senone-aware Adversarial Multi-task Training for Unsupervised Child to Adult Speech Adaptation
• Continual Novelty Detection
• More Parameters? No Thanks!
• Defeating Catastrophic Forgetting via Enhanced Orthogonal Weights Modification
• FFNB: Forgetting-Free Neural Blocks for Deep Continual Visual Learning
• Residual Continual Learning
• Exploring the Challenges towards Lifelong Fact Learning
• Knowledge-Adaptation Priors
• MyMigrationBot: A Cloud-based Facebook Social Chatbot for Migrant Populations
• Measuring Information Transfer in Neural Networks
• Association: Remind Your GAN not to Forget