Continual Learning in Low-rank Orthogonal Subspaces

Abstract

In continual learning (CL), a learner is faced with a sequence of tasks, arriving one after the other, and the goal is to remember all the tasks once the continual learning experience is finished. The prior art in CL uses episodic memory, parameter regularization or extensible network structures to reduce interference among tasks, but in the end, all the approaches learn different tasks in a joint vector space. We believe this invariably leads to interference among different tasks. We propose to learn tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Further, to keep the gradients of different tasks coming from these subspaces orthogonal to each other, we learn isometric mappings by posing network training as an optimization problem over the Stiefel manifold. To the best of our understanding, we report, for the first time, strong results over experience-replay baseline with and without memory on standard classification benchmarks in continual learning. The code is made publicly available.

arXiv commentThe paper is accepted at NeurIPS'20

References in corpus (22)

• Distilling the Knowledge in a Neural Network
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Lifelong Learning with A-GEM
• Experience Replay for Continual Learning
• On Tiny Episodic Memories in Continual Learning
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Reinforced Continual Learning
• Lifelong Learning with Dynamically Expandable Networks
• Full-Capacity Unitary Recurrent Neural Networks
• Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• Functional Regularisation for Continual Learning with Gaussian Processes
• Gradient based sample selection for online continual learning
• Learning to Continually Learn
• Automatically Composing Representation Transformations as a Means for Generalization
• Bounding and Counting Linear Regions of Deep Neural Networks
• Orthogonal Gradient Descent for Continual Learning
• Modular meta-learning
• Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform