Continual Deep Learning by Functional Regularisation of Memorable Past

Abstract

Continually learning new skills is important for intelligent systems, yet standard deep learning methods suffer from catastrophic forgetting of the past. Recent works address this with weight regularisation. Functional regularisation, although computationally expensive, is expected to perform better, but rarely does so in practice. In this paper, we fix this issue by using a new functional-regularisation approach that utilises a few memorable past examples crucial to avoid forgetting. By using a Gaussian Process formulation of deep networks, our approach enables training in weight-space while identifying both the memorable past and a functional prior. Our method achieves state-of-the-art performance on standard benchmarks and opens a new direction for life-long learning where regularisation and memory-based methods are naturally combined.

References in corpus (10)

• Overcoming catastrophic forgetting in neural networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Lifelong Learning with A-GEM
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Practical Deep Learning with Bayesian Principles
• Functional Regularisation for Continual Learning with Gaussian Processes
• Improving and Understanding Variational Continual Learning

Cited by in corpus (13)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Continual Learning and Catastrophic Forgetting
• Laplace Redux -- Effortless Bayesian Deep Learning
• Continual learning of longitudinal health records
• A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix
• Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
• Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning
• Improving predictions of Bayesian neural nets via local linearization
• On Sequential Bayesian Inference for Continual Learning
• Formalizing the Generalization-Forgetting Trade-off in Continual Learning
• Posterior Meta-Replay for Continual Learning
• Efficient Data-Parallel Continual Learning with Asynchronous Distributed Rehearsal Buffers
• Fast Adaptation with Linearized Neural Networks