Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting

Abstract

We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the weights. The Laplace approximation requires calculating the Hessian around a mode, which is typically intractable for modern architectures. In order to make our method scalable, we leverage recent block-diagonal Kronecker factored approximations to the curvature. Our algorithm achieves over 90% test accuracy across a sequence of 50 instantiations of the permuted MNIST dataset, substantially outperforming related methods for overcoming catastrophic forgetting.

arXiv comment13 pages, 6 figures

Cited by in corpus (60)

• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Towards Robust Evaluations of Continual Learning
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data
• Understanding the Role of Training Regimes in Continual Learning
• A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications
• Task Agnostic Continual Learning Using Online Variational Bayes
• Continual Deep Learning by Functional Regularisation of Memorable Past
• Overcoming Long-term Catastrophic Forgetting through Adversarial Neural Pruning and Synaptic Consolidation
• A Unifying Bayesian View of Continual Learning
• Continual Learning and Catastrophic Forgetting
• Recent Advances of Continual Learning in Computer Vision: An Overview
• Toward Understanding Catastrophic Forgetting in Continual Learning
• Improved Schemes for Episodic Memory-based Lifelong Learning
• Laplace Redux -- Effortless Bayesian Deep Learning
• Self-Supervised Training Enhances Online Continual Learning
• REMIND Your Neural Network to Prevent Catastrophic Forgetting
• Regularization Shortcomings for Continual Learning
• Attended Temperature Scaling: A Practical Approach for Calibrating Deep Neural Networks
• Measuring and regularizing networks in function space
• Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates
• Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint
• Online Class-Incremental Continual Learning with Adversarial Shapley Value
• Linear Mode Connectivity in Multitask and Continual Learning
• Reconciling meta-learning and continual learning with online mixtures of tasks
• Sentence Embedding Alignment for Lifelong Relation Extraction
• Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis
• Lifelong Learning of Compositional Structures
• Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
• Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting
• Example Forgetting: A Novel Approach to Explain and Interpret Deep Neural Networks in Seismic Interpretation
• Explaining Deep Models through Forgettable Learning Dynamics
• Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning
• Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning
• Continual Learning using a Bayesian Nonparametric Dictionary of Weight Factors
• On Sequential Bayesian Inference for Continual Learning
• Forgetful Active Learning with Switch Events: Efficient Sampling for Out-of-Distribution Data
• A Coordinate-Free Construction of Scalable Natural Gradient
• Understanding Continual Learning Settings with Data Distribution Drift Analysis
• Continual Model-Based Reinforcement Learning with Hypernetworks
• Unifying Regularisation Methods for Continual Learning
• Posterior Meta-Replay for Continual Learning
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Memory Efficient Class-Incremental Learning for Image Classification
• Learnable Uncertainty under Laplace Approximations
• A Deep Learning Framework for Lifelong Machine Learning
• Overcoming Catastrophic Forgetting by Soft Parameter Pruning
• GROWN: GRow Only When Necessary for Continual Learning
• Variational Auto-Regressive Gaussian Processes for Continual Learning
• Curriculum-Meta Learning for Order-Robust Continual Relation Extraction
• Hidden Markov Neural Networks
• A Conceptual Framework for Lifelong Learning
• Graph-Based Continual Learning
• ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
• Continual Learning in Deep Networks: an Analysis of the Last Layer
• How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolation
• Online Continual Learning in Image Classification: An Empirical Survey
• Kronecker Factorization for Preventing Catastrophic Forgetting in Large-scale Medical Entity Linking
• FFNB: Forgetting-Free Neural Blocks for Deep Continual Visual Learning