Lifelong Learning with Dynamically Expandable Networks

Abstract

We propose a novel deep network architecture for lifelong learning which we refer to as Dynamically Expandable Network (DEN), that can dynamically decide its network capacity as it trains on a sequence of tasks, to learn a compact overlapping knowledge sharing structure among tasks. DEN is efficiently trained in an online manner by performing selective retraining, dynamically expands network capacity upon arrival of each task with only the necessary number of units, and effectively prevents semantic drift by splitting/duplicating units and timestamping them. We validate DEN on multiple public datasets under lifelong learning scenarios, on which it not only significantly outperforms existing lifelong learning methods for deep networks, but also achieves the same level of performance as the batch counterparts with substantially fewer number of parameters. Further, the obtained network fine-tuned on all tasks obtained significantly better performance over the batch models, which shows that it can be used to estimate the optimal network structure even when all tasks are available in the first place.

11 pages, 6 figures

References in corpus (4)

• Overcoming catastrophic forgetting in neural networks
• Learning Structured Sparsity in Deep Neural Networks
• AdaNet: Adaptive Structural Learning of Artificial Neural Networks
• Nonparametric Neural Networks

Cited by in corpus (48)

• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• On Tiny Episodic Memories in Continual Learning
• Overcoming catastrophic forgetting with hard attention to the task
• Reinforced Continual Learning
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
• Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis
• Class-incremental Learning with Pre-allocated Fixed Classifiers
• Continual Learning in Low-rank Orthogonal Subspaces
• Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning
• Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractors
• Machine Learning for the Detection and Identification of Internet of Things (IoT) Devices: A Survey
• One Person, One Model, One World: Learning Continual User Representation without Forgetting
• Few-Shot Class-Incremental Learning
• Online Continual Learning via the Knowledge Invariant and Spread-out Properties
• ConDA: Continual Unsupervised Domain Adaptation
• Neural Architecture Search for Class-incremental Learning
• Lifelong Person Re-Identification via Adaptive Knowledge Accumulation
• Online Structured Meta-learning
• Self-Supervised Learning Aided Class-Incremental Lifelong Learning
• Supermasks in Superposition
• A Lifelong Learning Approach to Mobile Robot Navigation
• AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
• Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware
• Energy-Based Models for Continual Learning
• OpenLORIS-Object: A Robotic Vision Dataset and Benchmark for Lifelong Deep Learning
• Continual Learning Using Bayesian Neural Networks
• AdaFilter: Adaptive Filter Fine-tuning for Deep Transfer Learning
• Continual Learning Using Multi-view Task Conditional Neural Networks
• Meta Continual Learning via Dynamic Programming
• Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control
• GROWN: GRow Only When Necessary for Continual Learning
• Single-Net Continual Learning with Progressive Segmented Training (PST)
• Continual Learning via Bit-Level Information Preserving
• Incremental Concept Learning via Online Generative Memory Recall
• IROS 2019 Lifelong Robotic Vision Challenge -- Lifelong Object Recognition Report
• KSM: Fast Multiple Task Adaption via Kernel-wise Soft Mask Learning
• Graph-Based Continual Learning
• Unsupervised Class-Incremental Learning Through Confusion
• ZS-IL: Looking Back on Learned Experiences For Zero-Shot Incremental Learning
• Dynamic VAEs with Generative Replay for Continual Zero-shot Learning
• Continual learning using hash-routed convolutional neural networks
• TAG: Task-based Accumulated Gradients for Lifelong learning
• LIRA: Lifelong Image Restoration from Unknown Blended Distortions
• Shared and Private VAEs with Generative Replay for Continual Learning
• A Novel Approach to Lifelong Learning: The Plastic Support Structure
• Discriminative Distillation to Reduce Class Confusion in Continual Learning
• Class Incremental Online Streaming Learning
• Association: Remind Your GAN not to Forget