GROWN: GRow Only When Necessary for Continual Learning

Abstract

Catastrophic forgetting is a notorious issue in deep learning, referring to the fact that Deep Neural Networks (DNN) could forget the knowledge about earlier tasks when learning new tasks. To address this issue, continual learning has been developed to learn new tasks sequentially and perform knowledge transfer from the old tasks to the new ones without forgetting. While recent structure-based learning methods show the capability of alleviating the forgetting problem, these methods start from a redundant full-size network and require a complex learning process to gradually grow-and-prune or search the network structure for each task, which is inefficient. To address this problem and enable efficient network expansion for new tasks, we first develop a learnable sparse growth method eliminating the additional pruning/searching step in previous structure-based methods. Building on this learnable sparse growth method, we then propose GROWN, a novel end-to-end continual learning framework to dynamically grow the model only when necessary. Different from all previous structure-based methods, GROWN starts from a small seed network, instead of a full-sized one. We validate GROWN on multiple datasets against state-of-the-art methods, which shows superior performance in both accuracy and model size. For example, we achieve 1.0\% accuracy gain on average compared to the current SOTA results on CIFAR-100 Superclass 20 tasks setting.

arXiv comment8 pages. arXiv admin note: text overlap with arXiv:2007.07617 by other authors

References in corpus (13)

• Overcoming catastrophic forgetting in neural networks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Progress & Compress: A scalable framework for continual learning
• Continual Learning with Deep Generative Replay
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Lifelong Learning with Dynamically Expandable Networks
• Compacting, Picking and Growing for Unforgetting Continual Learning
• Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
• Efficient Continual Learning in Neural Networks with Embedding Regularization
• Efficient Continual Learning with Modular Networks and Task-Driven Priors
• Growing Efficient Deep Networks by Structured Continuous Sparsification