Continual Learning via Neural Pruning

Abstract

We introduce Continual Learning via Neural Pruning (CLNP), a new method aimed at lifelong learning in fixed capacity models based on neuronal model sparsification. In this method, subsequent tasks are trained using the inactive neurons and filters of the sparsified network and cause zero deterioration to the performance of previous tasks. In order to deal with the possible compromise between model sparsity and performance, we formalize and incorporate the concept of graceful forgetting: the idea that it is preferable to suffer a small amount of forgetting in a controlled manner if it helps regain network capacity and prevents uncontrolled loss of performance during the training of future tasks. CLNP also provides simple continual learning diagnostic tools in terms of the number of free neurons left for the training of future tasks as well as the number of neurons that are being reused. In particular, we see in experiments that CLNP verifies and automatically takes advantage of the fact that the features of earlier layers are more transferable. We show empirically that CLNP leads to significantly improved results over current weight elasticity based methods.

12 pages, 5 figures, 3 tables

References in corpus (7)

• Overcoming catastrophic forgetting in neural networks
• How transferable are features in deep neural networks?
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• The State of Sparsity in Deep Neural Networks
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Power Law in Sparsified Deep Neural Networks

Cited by in corpus (28)

• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Continual Learning for Recurrent Neural Networks: an Empirical Evaluation
• SpaceNet: Make Free Space For Continual Learning
• Task Agnostic Continual Learning via Meta Learning
• Continual Learning and Catastrophic Forgetting
• Dynamically Expandable Graph Convolution for Streaming Recommendation
• Improving and Understanding Variational Continual Learning
• Continual Learning with Node-Importance based Adaptive Group Sparse Regularization
• AdaptCL: Adaptive Continual Learning for Tackling Heterogeneity in Sequential Datasets
• Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
• Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation
• Cooperative data-driven modeling
• One Person, One Model, One World: Learning Continual User Representation without Forgetting
• Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
• Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation
• Supermasks in Superposition
• Unifying Regularisation Methods for Continual Learning
• Bayesian Structure Adaptation for Continual Learning
• Learn-Prune-Share for Lifelong Learning
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• Continual Learning: Forget-free Winning Subnetworks for Video Representations
• A Conceptual Framework for Lifelong Learning
• Tuned Compositional Feature Replays for Efficient Stream Learning
• A Biologically Plausible Audio-Visual Integration Model for Continual Learning
• Structured Ensembles: an Approach to Reduce the Memory Footprint of Ensemble Methods
• Mixture-of-Variational-Experts for Continual Learning
• Multi-Scale Weight Sharing Network for Image Recognition
• Emergent Structures and Lifetime Structure Evolution in Artificial Neural Networks