SpaceNet: Make Free Space For Continual Learning

Abstract

The continual learning (CL) paradigm aims to enable neural networks to learn tasks continually in a sequential fashion. The fundamental challenge in this learning paradigm is catastrophic forgetting previously learned tasks when the model is optimized for a new task, especially when their data is not accessible. Current architectural-based methods aim at alleviating the catastrophic forgetting problem but at the expense of expanding the capacity of the model. Regularization-based methods maintain a fixed model capacity; however, previous studies showed the huge performance degradation of these methods when the task identity is not available during inference (e.g. class incremental learning scenario). In this work, we propose a novel architectural-based method referred as SpaceNet for class incremental learning scenario where we utilize the available fixed capacity of the model intelligently. SpaceNet trains sparse deep neural networks from scratch in an adaptive way that compresses the sparse connections of each task in a compact number of neurons. The adaptive training of the sparse connections results in sparse representations that reduce the interference between the tasks. Experimental results show the robustness of our proposed method against catastrophic forgetting old tasks and the efficiency of SpaceNet in utilizing the available capacity of the model, leaving space for more tasks to be learned. In particular, when SpaceNet is tested on the well-known benchmarks for CL: split MNIST, split Fashion-MNIST, and CIFAR-10/100, it outperforms regularization-based methods by a big performance gap. Moreover, it achieves better performance than architectural-based methods without model expansion and achieved comparable results with rehearsal-based methods, while offering a huge memory reduction.

arXiv commentPublished in Neurocomputing Journal

References in corpus (22)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Wide Residual Networks
• Continual Learning Through Synaptic Intelligence
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Three scenarios for continual learning
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• Progress & Compress: A scalable framework for continual learning
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Towards Robust Evaluations of Continual Learning
• Deep Rewiring: Training very sparse deep networks
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Rigging the Lottery: Making All Tickets Winners
• Continual Learning via Neural Pruning
• A topological insight into restricted Boltzmann machines
• Efficient Continual Learning in Neural Networks with Embedding Regularization
• Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
• Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data
• LCA: Loss Change Allocation for Neural Network Training
• Topological Insights into Sparse Neural Networks

Cited by in corpus (9)

• Continual Learning for Recurrent Neural Networks: an Empirical Evaluation
• Continual Prune-and-Select: Class-incremental learning with specialized subnetworks
• Cooperative data-driven modeling
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• Powerpropagation: A sparsity inducing weight reparameterisation
• Learning Invariant Representation for Continual Learning
• Class-Incremental Experience Replay for Continual Learning under Concept Drift
• A Practical Guide to Streaming Continual Learning
• Toward industrial use of continual learning : new metrics proposal for class incremental learning