Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control
arXiv:1907.13322
Abstract
To address the issue of catastrophic forgetting in neural networks, we propose a novel, simple, and effective solution called neuron-level plasticity control (NPC). While learning a new task, the proposed method preserves the knowledge for the previous tasks by controlling the plasticity of the network at the neuron level. NPC estimates the importance value of each neuron and consolidates important \textit{neurons} by applying lower learning rates, rather than restricting individual connection weights to stay close to certain values. The experimental results on the incremental MNIST (iMNIST) and incremental CIFAR100 (iCIFAR100) datasets show that neuron-level consolidation is substantially more effective compared to the connection-level consolidation approaches.
arXiv comment8 pages
References in corpus (13)
- Overcoming catastrophic forgetting in neural networks
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- Continual Learning Through Synaptic Intelligence
- PathNet: Evolution Channels Gradient Descent in Super Neural Networks
- Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
- Overcoming Catastrophic Forgetting by Incremental Moment Matching
- Overcoming catastrophic forgetting with hard attention to the task
- Lifelong Learning with Dynamically Expandable Networks
- An Entropy-based Pruning Method for CNN Compression
- Deep Generative Dual Memory Network for Continual Learning
- Selfless Sequential Learning
- Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting