Unifying Regularisation Methods for Continual Learning
arXiv:2006.06357
Abstract
Continual Learning addresses the challenge of learning a number of different tasks sequentially. The goal of maintaining knowledge of earlier tasks without re-accessing them starkly conflicts with standard SGD training for artificial neural networks. An influential method to tackle this problem without storing old data are so-called regularisation approaches. They measure the importance of each parameter for solving a given task and subsequently protect important parameters from large changes. In the literature, three ways to measure parameter importance have been put forward and they have inspired a large body of follow-up work. Here, we present strong theoretical and empirical evidence that these three methods, Elastic Weight Consolidation (EWC), Synaptic Intelligence (SI) and Memory Aware Synapses (MAS), are surprisingly similar and are all linked to the same theoretical quantity. Concretely, we show that, despite stemming from very different motivations, both SI and MAS approximate the square root of the Fisher Information, with the Fisher being the theoretically justified basis of EWC. Moreover, we show that for SI the relation to the Fisher -- and in fact its performance -- is due to a previously unknown bias. On top of uncovering unknown similarities and unifying regularisation approaches, we also demonstrate that our insights enable practical performance improvements for large batch training.
References in corpus (22)
- Overcoming catastrophic forgetting in neural networks
- Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
- Continual Learning Through Synaptic Intelligence
- An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
- PathNet: Evolution Channels Gradient Descent in Super Neural Networks
- Three scenarios for continual learning
- FearNet: Brain-Inspired Model for Incremental Learning
- New insights and perspectives on the natural gradient method
- Variational Continual Learning
- Three Factors Influencing Minima in SGD
- Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
- Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
- Towards Robust Evaluations of Continual Learning
- On Quadratic Penalties in Elastic Weight Consolidation
- Continual Learning via Neural Pruning
- Understanding the Role of Training Regimes in Continual Learning
- Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
- Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
- On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
- Improving and Understanding Variational Continual Learning
- LCA: Loss Change Allocation for Neural Network Training
- Generalized Variational Continual Learning