Layerwise Optimization by Gradient Decomposition for Continual Learning

Abstract

Deep neural networks achieve state-of-the-art and sometimes super-human performance across various domains. However, when learning tasks sequentially, the networks easily forget the knowledge of previous tasks, known as "catastrophic forgetting". To achieve the consistencies between the old tasks and the new task, one effective solution is to modify the gradient for update. Previous methods enforce independent gradient constraints for different tasks, while we consider these gradients contain complex information, and propose to leverage inter-task information by gradient decomposition. In particular, the gradient of an old task is decomposed into a part shared by all old tasks and a part specific to that task. The gradient for update should be close to the gradient of the new task, consistent with the gradients shared by all old tasks, and orthogonal to the space spanned by the gradients specific to the old tasks. In this way, our approach encourages common knowledge consolidation without impairing the task-specific knowledge. Furthermore, the optimization is performed for the gradients of each layer separately rather than the concatenation of all gradients as in previous works. This effectively avoids the influence of the magnitude variation of the gradients in different layers. Extensive experiments validate the effectiveness of both gradient-decomposed optimization and layer-wise updates. Our proposed method achieves state-of-the-art results on various benchmarks of continual learning.

arXiv commentcvpr2021

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• An overview of gradient descent optimization algorithms
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• Three scenarios for continual learning
• Large Batch Training of Convolutional Networks
• Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification
• On Tiny Episodic Memories in Continual Learning
• Variational Continual Learning
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Less-forgetting Learning in Deep Neural Networks
• A PAC-Bayesian bound for Lifelong Learning
• Task Agnostic Continual Learning Using Online Variational Bayes
• End-to-End Incremental Learning
• Towards Training Recurrent Neural Networks for Lifelong Learning
• Gradient Regularized Contrastive Learning for Continual Domain Adaptation
• Complementary Relation Contrastive Distillation