Wide Neural Networks Forget Less Catastrophically

Abstract

A primary focus area in continual learning research is alleviating the "catastrophic forgetting" problem in neural networks by designing new algorithms that are more robust to the distribution shifts. While the recent progress in continual learning literature is encouraging, our understanding of what properties of neural networks contribute to catastrophic forgetting is still limited. To address this, instead of focusing on continual learning algorithms, in this work, we focus on the model itself and study the impact of "width" of the neural network architecture on catastrophic forgetting, and show that width has a surprisingly significant effect on forgetting. To explain this effect, we study the learning dynamics of the network from various perspectives such as gradient orthogonality, sparsity, and lazy training regime. We provide potential explanations that are consistent with the empirical results across different architectures and continual learning benchmarks.

arXiv commentICML 2022

References in corpus (25)

• Overcoming catastrophic forgetting in neural networks
• On the difficulty of training Recurrent Neural Networks
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Lifelong Learning with A-GEM
• On Exact Computation with an Infinitely Wide Neural Net
• Experience Replay for Continual Learning
• On Tiny Episodic Memories in Continual Learning
• Variational Continual Learning
• On Lazy Training in Differentiable Programming
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Reinforced Continual Learning
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Understanding the Role of Training Regimes in Continual Learning
• Functional Regularisation for Continual Learning with Gaussian Processes
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
• Learning to Continually Learn
• Automatically Composing Representation Transformations as a Means for Generalization
• Toward Understanding Catastrophic Forgetting in Continual Learning
• Modular meta-learning
• Supermasks in Superposition