Compacting, Picking and Growing for Unforgetting Continual Learning

Abstract

Continual lifelong learning is essential to many applications. In this paper, we propose a simple but effective approach to continual deep learning. Our approach leverages the principles of deep model compression, critical weights selection, and progressive networks expansion. By enforcing their integration in an iterative manner, we introduce an incremental learning method that is scalable to the number of sequential tasks in a continual learning process. Our approach is easy to implement and owns several favorable characteristics. First, it can avoid forgetting (i.e., learn new tasks while remembering all previous tasks). Second, it allows model expansion but can maintain the model compactness when handling sequential tasks. Besides, through our compaction and selection/expansion mechanism, we show that the knowledge accumulated through learning previous tasks is helpful to build a better model for the new tasks compared to training the models independently with tasks. Experimental results show that our approach can incrementally learn a deep model tackling multiple tasks without forgetting, while the model compactness is maintained with the performance more satisfiable than individual task training.

To appear in NeurIPS 2019

References in corpus (11)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Overcoming catastrophic forgetting in neural networks
• AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Incremental Classifier Learning with Generative Adversarial Networks
• A comprehensive, application-oriented study of catastrophic forgetting in DNNs
• Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature
• HOUDINI: Lifelong Learning as Program Synthesis
• On the role of neurogenesis in overcoming catastrophic forgetting

Cited by in corpus (15)

• Facial expression and attributes recognition based on multi-task learning of lightweight neural networks
• Continual Learning for Recurrent Neural Networks: an Empirical Evaluation
• Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis
• Continual Learning and Catastrophic Forgetting
• Federated Continual Learning with Weighted Inter-client Transfer
• Class Gradient Projection For Continual Learning
• Generalizing MLPs With Dropouts, Batch Normalization, and Skip Connections
• Leveraging Old Knowledge to Continually Learn New Classes in Medical Images
• Continual Learning using a Bayesian Nonparametric Dictionary of Weight Factors
• GROWN: GRow Only When Necessary for Continual Learning
• KSM: Fast Multiple Task Adaption via Kernel-wise Soft Mask Learning
• Visually Grounded Continual Language Learning with Selective Specialization
• Discriminative Distillation to Reduce Class Confusion in Continual Learning
• Compression-aware Continual Learning using Singular Value Decomposition
• Continual Learning for Task-oriented Dialogue System with Iterative Network Pruning, Expanding and Masking