Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting

Abstract

In this paper we propose an approach to avoiding catastrophic forgetting in sequential task learning scenarios. Our technique is based on a network reparameterization that approximately diagonalizes the Fisher Information Matrix of the network parameters. This reparameterization takes the form of a factorized rotation of parameter space which, when used in conjunction with Elastic Weight Consolidation (which assumes a diagonal Fisher Information Matrix), leads to significantly better performance on lifelong learning of sequential tasks. Experimental results on the MNIST, CIFAR-100, CUB-200 and Stanford-40 datasets demonstrate that we significantly improve the results of standard elastic weight consolidation, and that we obtain competitive results when compared to other state-of-the-art in lifelong learning without forgetting.

arXiv commentAccepted at ICPR'18. First two authors contributed equally

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Overcoming catastrophic forgetting with hard attention to the task
• Continual Learning with Deep Generative Replay
• Natural Neural Networks
• Diagonal Rescaling For Neural Networks

Cited by in corpus (15)

• Selfless Sequential Learning
• Metric Learning for Novelty and Anomaly Detection
• Memory Replay GANs: learning to generate images from new categories without forgetting
• Class-incremental Learning via Deep Model Consolidation
• RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning
• Few-Shot Class-Incremental Learning
• Carousel Memory: Rethinking the Design of Episodic Memory for Continual Learning
• Towards Training Recurrent Neural Networks for Lifelong Learning
• Incremental Learning with Maximum Entropy Regularization: Rethinking Forgetting and Intransigence
• Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control
• Regularize, Expand and Compress: Multi-task based Lifelong Learning via NonExpansive AutoML
• Local learning rules to attenuate forgetting in neural networks
• Incremental Embedding Learning via Zero-Shot Translation
• Bilevel Continual Learning
• Online Continual Learning in Image Classification: An Empirical Survey