Group and Exclusive Sparse Regularization-based Continual Learning of CNNs

Abstract

We present a regularization-based approach for continual learning (CL) of fixed capacity convolutional neural networks (CNN) that does not suffer from the problem of catastrophic forgetting when learning multiple tasks sequentially. This method referred to as Group and Exclusive Sparsity based Continual Learning (GESCL) avoids forgetting of previous tasks by ensuring the stability of the CNN via a stability regularization term, which prevents filters detected as important for past tasks to deviate too much when learning a new task. On top of that, GESCL makes the network plastic via a plasticity regularization term that leverage the over-parameterization of CNNs to efficiently sparsify the network and tunes unimportant filters making them relevant for future tasks. Doing so, GESCL deals with significantly less parameters and computation compared to CL approaches that either dynamically expand the network or memorize past tasks' data. Experiments on popular CL vision benchmarks show that GESCL leads to significant improvements over state-of-the-art method in terms of overall CL performance, as measured by classification accuracy as well as in terms of avoiding catastrophic forgetting.

12 pages, Canadian Artificial Intelligence Association (CAIAC)

References in corpus (11)

• Overcoming catastrophic forgetting in neural networks
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Continual Learning Through Synaptic Intelligence
• Gradient Episodic Memory for Continual Learning
• Overcoming catastrophic forgetting with hard attention to the task
• Understanding the Role of Training Regimes in Continual Learning
• Continual Learning with Node-Importance based Adaptive Group Sparse Regularization
• Learning where to learn: Gradient sparsity in meta and continual learning
• Provable Filter Pruning for Efficient Neural Networks
• Scalable and Order-robust Continual Learning with Additive Parameter Decomposition
• Continual Learning via Local Module Composition