Continual Learning and Catastrophic Forgetting
arXiv:2403.05175 · doi:10.1016/B978-0-443-15754-7.00073-0
Abstract
This book chapter delves into the dynamics of continual learning, which is the process of incrementally learning from a non-stationary stream of data. Although continual learning is a natural skill for the human brain, it is very challenging for artificial neural networks. An important reason is that, when learning something new, these networks tend to quickly and drastically forget what they had learned before, a phenomenon known as catastrophic forgetting. Especially in the last decade, continual learning has become an extensively studied topic in deep learning. This book chapter reviews the insights that this field has generated.
arXiv commentPreprint of a book chapter; 21 pages, 4 figures
References in corpus (38)
- Overcoming catastrophic forgetting in neural networks
- Prototypical Networks for Few-shot Learning
- A continual learning survey: Defying forgetting in classification tasks
- Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence
- Efficient Lifelong Learning with A-GEM
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Experience Replay for Continual Learning
- Progress & Compress: A scalable framework for continual learning
- Variational Continual Learning
- Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
- Continual Learning of Context-dependent Processing in Neural Networks
- A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
- Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
- Compacting, Picking and Growing for Unforgetting Continual Learning
- A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
- Understanding the Role of Training Regimes in Continual Learning
- Continual Learning via Neural Pruning
- Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligence
- Functional Regularisation for Continual Learning with Gaussian Processes
- Synaptic metaplasticity in binarized neural networks
- Online Continual Learning with Maximally Interfered Retrieval
- Continual Deep Learning by Functional Regularisation of Memorable Past
- Gradient-based Editing of Memory Examples for Online Task-free Continual Learning
- Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System
- Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning
- A Theoretical Study on Solving Continual Learning
- Continual Learning: Applications and the Road Forward
- How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition
- Continual Learning with Foundation Models: An Empirical Study of Latent Replay
- On Sequential Bayesian Inference for Continual Learning
- Continual Learning via Sequential Function-Space Variational Inference
- BiRT: Bio-inspired Replay in Vision Transformers for Continual Learning
- TACOS: Task Agnostic Continual Learning in Spiking Neural Networks
- Learn the Time to Learn: Replay Scheduling in Continual Learning
- Class-Incremental Learning with Repetition
- Prediction Error-based Classification for Class-Incremental Learning
- Continual Learning in the Presence of Repetition
- Two Complementary Perspectives to Continual Learning: Ask Not Only What to Optimize, But Also How