Understanding the Role of Training Regimes in Continual Learning

Abstract

Catastrophic forgetting affects the training of neural networks, limiting their ability to learn multiple tasks sequentially. From the perspective of the well established plasticity-stability dilemma, neural networks tend to be overly plastic, lacking the stability necessary to prevent the forgetting of previous knowledge, which means that as learning progresses, networks tend to forget previously seen tasks. This phenomenon coined in the continual learning literature, has attracted much attention lately, and several families of approaches have been proposed with different degrees of success. However, there has been limited prior work extensively analyzing the impact that different training regimes -- learning rate, batch size, regularization method-- can have on forgetting. In this work, we depart from the typical approach of altering the learning algorithm to improve stability. Instead, we hypothesize that the geometrical properties of the local minima found for each task play an important role in the overall degree of forgetting. In particular, we study the effect of dropout, learning rate decay, and batch size, on forming training regimes that widen the tasks' local minima and consequently, on helping it not to forget catastrophically. Our study provides practical insights to improve stability via simple yet effective techniques that outperform alternative baselines.

References in corpus (26)

• Overcoming catastrophic forgetting in neural networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Don't Decay the Learning Rate, Increase the Batch Size
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Revisiting Small Batch Training for Deep Neural Networks
• On Tiny Episodic Memories in Continual Learning
• Variational Continual Learning
• Continual Learning with Deep Generative Replay
• Qualitatively characterizing neural network optimization problems
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Towards Robust Evaluations of Continual Learning
• Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
• Functional Regularisation for Continual Learning with Gaussian Processes
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
• Gradient based sample selection for online continual learning
• Learning to Continually Learn
• The large learning rate phase of deep learning: the catapult mechanism
• Task Agnostic Continual Learning via Meta Learning
• On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
• Orthogonal Gradient Descent for Continual Learning
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
• Variational Prototype Replays for Continual Learning

Cited by in corpus (19)

• Continual Learning and Catastrophic Forgetting
• Recent Advances of Continual Learning in Computer Vision: An Overview
• Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning
• The Effectiveness of Memory Replay in Large Scale Continual Learning
• A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix
• Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint
• Linear Mode Connectivity in Multitask and Continual Learning
• Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Towards Generalized and Incremental Few-Shot Object Detection
• Formalizing the Generalization-Forgetting Trade-off in Continual Learning
• Unifying Regularisation Methods for Continual Learning
• Wide Neural Networks Forget Less Catastrophically
• Efficient Data-Parallel Continual Learning with Asynchronous Distributed Rehearsal Buffers
• Group and Exclusive Sparse Regularization-based Continual Learning of CNNs
• Tuned Compositional Feature Replays for Efficient Stream Learning
• Does the Adam Optimizer Exacerbate Catastrophic Forgetting?
• Continual Novelty Detection
• TAG: Task-based Accumulated Gradients for Lifelong learning