Frosting Weights for Better Continual Training

Abstract

Training a neural network model can be a lifelong learning process and is a computationally intensive one. A severe adverse effect that may occur in deep neural network models is that they can suffer from catastrophic forgetting during retraining on new data. To avoid such disruptions in the continuous learning, one appealing property is the additive nature of ensemble models. In this paper, we propose two generic ensemble approaches, gradient boosting and meta-learning, to solve the catastrophic forgetting problem in tuning pre-trained neural network models.

References in corpus (32)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• How transferable are features in deep neural networks?
• An Overview of Multi-Task Learning in Deep Neural Networks
• Progressive Neural Networks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• A Survey on Multi-Task Learning
• Efficient Lifelong Learning with A-GEM
• Gradient Episodic Memory for Continual Learning
• On Tiny Episodic Memories in Continual Learning
• FearNet: Brain-Inspired Model for Incremental Learning
• Progress & Compress: A scalable framework for continual learning
• Variational Continual Learning
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Slimmable Neural Networks
• Which Tasks Should Be Learned Together in Multi-task Learning?
• Overcoming catastrophic forgetting with hard attention to the task
• Reinforced Continual Learning
• Deep Generative Dual Memory Network for Continual Learning
• Continual Learning in Generative Adversarial Nets
• Online Continual Learning with Maximally Interfered Retrieval
• Selfless Sequential Learning
• Task Agnostic Continual Learning Using Online Variational Bayes
• Generative Adversarial Network Training is a Continual Learning Problem
• Improving and Understanding Variational Continual Learning
• Pseudo-task Augmentation: From Deep Multitask Learning to Intratask Sharing---and Back
• SupportNet: solving catastrophic forgetting in class incremental learning with support data
• On Catastrophic Forgetting and Mode Collapse in Generative Adversarial Networks
• Benchmark Environments for Multitask Learning in Continuous Domains
• Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification