Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent

Abstract

In Continual Learning settings, deep neural networks are prone to Catastrophic Forgetting. Orthogonal Gradient Descent was proposed to tackle the challenge. However, no theoretical guarantees have been proven yet. We present a theoretical framework to study Continual Learning algorithms in the Neural Tangent Kernel regime. This framework comprises closed form expression of the model through tasks and proxies for Transfer Learning, generalisation and tasks similarity. In this framework, we prove that OGD is robust to Catastrophic Forgetting then derive the first generalisation bound for SGD and OGD for Continual Learning. Finally, we study the limits of this framework in practice for OGD and highlight the importance of the Neural Tangent Kernel variation for Continual Learning with OGD.

References in corpus (26)

• Overcoming catastrophic forgetting in neural networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Efficient Lifelong Learning with A-GEM
• Three scenarios for continual learning
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Overcoming catastrophic forgetting with hard attention to the task
• Generative replay with feedback connections as a general strategy for continual learning
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• Gradient Surgery for Multi-Task Learning
• Don't forget, there is more than forgetting: new metrics for Continual Learning
• Understanding the Role of Training Regimes in Continual Learning
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• Continual Deep Learning by Functional Regularisation of Memorable Past
• Orthogonal Gradient Descent for Continual Learning
• Toward Understanding Catastrophic Forgetting in Continual Learning
• Continual Learning with Node-Importance based Adaptive Group Sparse Regularization
• Is Long Horizon Reinforcement Learning More Difficult Than Short Horizon Reinforcement Learning?
• Incremental Learning-to-Learn with Statistical Guarantees
• Towards Understanding the Transferability of Deep Representations
• Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations
• A Sample Complexity Separation between Non-Convex and Convex Meta-Learning

Cited by in corpus (7)

• Gradient Projection Memory for Continual Learning
• A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix
• Formalizing the Generalization-Forgetting Trade-off in Continual Learning
• TAG: Task-based Accumulated Gradients for Lifelong learning
• Provable Lifelong Learning of Representations
• Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks
• Defeating Catastrophic Forgetting via Enhanced Orthogonal Weights Modification