On Quadratic Penalties in Elastic Weight Consolidation

Abstract

Elastic weight consolidation (EWC, Kirkpatrick et al, 2017) is a novel algorithm designed to safeguard against catastrophic forgetting in neural networks. EWC can be seen as an approximation to Laplace propagation (Eskin et al, 2004), and this view is consistent with the motivation given by Kirkpatrick et al (2017). In this note, I present an extended derivation that covers the case when there are more than two tasks. I show that the quadratic penalties in EWC are inconsistent with this derivation and might lead to double-counting data from earlier tasks.

References in corpus (5)

• Overcoming catastrophic forgetting in neural networks
• Expectation Propagation for approximate Bayesian inference
• Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (2001)
• New insights and perspectives on the natural gradient method
• On Quadratic Penalties in Elastic Weight Consolidation

Cited by in corpus (24)

• A continual learning survey: Defying forgetting in classification tasks
• Three scenarios for continual learning
• Generative replay with feedback connections as a general strategy for continual learning
• Towards Robust Evaluations of Continual Learning
• Adaptive Extreme Edge Computing for Wearable Devices
• On Quadratic Penalties in Elastic Weight Consolidation
• Autonomous Deep Learning: Continual Learning Approach for Dynamic Environments
• Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis
• Efficient Continual Learning in Neural Networks with Embedding Regularization
• CLeaR: An Adaptive Continual Learning Framework for Regression Tasks
• Toward Understanding Catastrophic Forgetting in Continual Learning
• Laplace Redux -- Effortless Bayesian Deep Learning
• Continual World: A Robotic Benchmark For Continual Reinforcement Learning
• Continual Learning in Neural Networks
• Continual Learning in Recurrent Neural Networks
• Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting
• Unifying Regularisation Methods for Continual Learning
• Lifelong Learning using Eigentasks: Task Separation, Skill Acquisition, and Selective Transfer
• A Combinatorial Perspective on Transfer Learning
• Posterior Meta-Replay for Continual Learning
• Stabilizing Elastic Weight Consolidation method in practical ML tasks and using weight importances for neural network pruning
• Continual Learning With Quasi-Newton Methods
• DIRA: Dynamic Domain Incremental Regularised Adaptation
• Self-Net: Lifelong Learning via Continual Self-Modeling