Wide Neural Networks Forget Less Catastrophically

Abstract

A primary focus area in continual learning research is alleviating the "catastrophic forgetting" problem in neural networks by designing new algorithms that are more robust to the distribution shifts. While the recent progress in continual learning literature is encouraging, our understanding of what properties of neural networks contribute to catastrophic forgetting is still limited. To address this, instead of focusing on continual learning algorithms, in this work, we focus on the model itself and study the impact of "width" of the neural network architecture on catastrophic forgetting, and show that width has a surprisingly significant effect on forgetting. To explain this effect, we study the learning dynamics of the network from various perspectives such as gradient orthogonality, sparsity, and lazy training regime. We provide potential explanations that are consistent with the empirical results across different architectures and continual learning benchmarks.

ICML 2022

References in corpus (17)

• On the difficulty of training Recurrent Neural Networks
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Lifelong Learning with A-GEM
• On Exact Computation with an Infinitely Wide Neural Net
• Experience Replay for Continual Learning
• On Tiny Episodic Memories in Continual Learning
• On Lazy Training in Differentiable Programming
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Understanding the Role of Training Regimes in Continual Learning
• Functional Regularisation for Continual Learning with Gaussian Processes
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
• Learning to Continually Learn
• Toward Understanding Catastrophic Forgetting in Continual Learning
• Supermasks in Superposition