Linear Mode Connectivity in Multitask and Continual Learning

Abstract

Continual (sequential) training and multitask (simultaneous) training are often attempting to solve the same overall objective: to find a solution that performs well on all considered tasks. The main difference is in the training regimes, where continual learning can only have access to one task at a time, which for neural networks typically leads to catastrophic forgetting. That is, the solution found for a subsequent task does not perform well on the previous ones anymore. However, the relationship between the different minima that the two training regimes arrive at is not well understood. What sets them apart? Is there a local structure that could explain the difference in performance achieved by the two different schemes? Motivated by recent work showing that different minima of the same task are typically connected by very simple curves of low error, we investigate whether multitask and continual solutions are similarly connected. We empirically find that indeed such connectivity can be reliably achieved and, more interestingly, it can be done by a linear path, conditioned on having the same initialization for both. We thoroughly analyze this observation and discuss its significance for the continual learning process. Furthermore, we exploit this finding to propose an effective algorithm that constrains the sequentially learned minima to behave as the multitask solution. We show that our method outperforms several state of the art continual learning algorithms on various vision benchmarks.

References in corpus (36)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Overcoming catastrophic forgetting in neural networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Efficient Lifelong Learning with A-GEM
• Similarity of Neural Network Representations Revisited
• Experience Replay for Continual Learning
• Deep Ensembles: A Loss Landscape Perspective
• On Tiny Episodic Memories in Continual Learning
• Variational Continual Learning
• Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Measuring Catastrophic Forgetting in Neural Networks
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Towards Robust Evaluations of Continual Learning
• Essentially No Barriers in Neural Network Energy Landscape
• Understanding the Role of Training Regimes in Continual Learning
• Functional Regularisation for Continual Learning with Gaussian Processes
• Continual Unsupervised Representation Learning
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
• Gradient based sample selection for online continual learning
• Learning to Continually Learn
• Task Agnostic Continual Learning via Meta Learning
• Orthogonal Gradient Descent for Continual Learning
• Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
• Toward Understanding Catastrophic Forgetting in Continual Learning
• Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
• Emergent properties of the local geometry of neural loss landscapes
• Few-Shot Self Reminder to Overcome Catastrophic Forgetting
• Variational Prototype Replays for Continual Learning
• Overcoming Catastrophic Interference by Conceptors
• Supermasks in Superposition