Orthogonal Gradient Descent for Continual Learning

Abstract

Neural networks are achieving state of the art and sometimes super-human performance on learning tasks across a variety of domains. Whenever these problems require learning in a continual or sequential manner, however, neural networks suffer from the problem of catastrophic forgetting; they forget how to solve previous tasks after being trained on a new task, despite having the essential capacity to solve both tasks if they were trained on both simultaneously. In this paper, we propose to address this issue from a parameter space perspective and study an approach to restrict the direction of the gradient updates to avoid forgetting previously-learned data. We present the Orthogonal Gradient Descent (OGD) method, which accomplishes this goal by projecting the gradients from new tasks onto a subspace in which the neural network output on previous task does not change and the projected gradient is still in a useful direction for learning the new task. Our approach utilizes the high capacity of a neural network more efficiently and does not require storing the previously learned data that might raise privacy concerns. Experiments on common benchmarks reveal the effectiveness of the proposed OGD method.

References in corpus (40)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• FitNets: Hints for Thin Deep Nets
• A continual learning survey: Defying forgetting in classification tasks
• Continual Learning Through Synaptic Intelligence
• Towards Federated Learning at Scale: System Design
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Experience Replay for Continual Learning
• Progress & Compress: A scalable framework for continual learning
• Variational Continual Learning
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Measuring Catastrophic Forgetting in Neural Networks
• Towards Robust Evaluations of Continual Learning
• Split learning for health: Distributed deep learning without sharing raw patient data
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Deep Generative Dual Memory Network for Continual Learning
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
• Don't forget, there is more than forgetting: new metrics for Continual Learning
• Neurogenesis Deep Learning
• Functional Regularisation for Continual Learning with Gaussian Processes
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
• Task Agnostic Continual Learning via Meta Learning
• A Unifying Bayesian View of Continual Learning
• Toward Understanding Catastrophic Forgetting in Continual Learning
• Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalization
• Stochastic Gradient/Mirror Descent: Minimax Optimality and Implicit Regularization
• Differentially-Private "Draw and Discard" Machine Learning
• Meta Continual Learning
• Continual Reinforcement Learning with Complex Synapses
• Few-Shot Self Reminder to Overcome Catastrophic Forgetting
• Variational Prototype Replays for Continual Learning
• Attention-Based Structural-Plasticity
• Privacy-preserving Stochastic Gradual Learning
• Cross-View Policy Learning for Street Navigation
• Continual and Multi-task Reinforcement Learning With Shared Episodic Memory

Cited by in corpus (10)

• Gradient Surgery for Multi-Task Learning
• Understanding the Role of Training Regimes in Continual Learning
• Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis
• Continual Learning in Low-rank Orthogonal Subspaces
• Meta-Consolidation for Continual Learning
• Linear Mode Connectivity in Multitask and Continual Learning
• Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
• Routing Networks with Co-training for Continual Learning
• Continual Learning with Deep Artificial Neurons
• Federated Learning with Fair Averaging