A study on the plasticity of neural networks

Abstract

One aim shared by multiple settings, such as continual learning or transfer learning, is to leverage previously acquired knowledge to converge faster on the current task. Usually this is done through fine-tuning, where an implicit assumption is that the network maintains its plasticity, meaning that the performance it can reach on any given task is not affected negatively by previously seen tasks. It has been observed recently that a pretrained model on data from the same distribution as the one it is fine-tuned on might not reach the same generalisation as a freshly initialised one. We build and extend this observation, providing a hypothesis for the mechanics behind it. We discuss the implication of losing plasticity for continual learning which heavily relies on optimising pretrained models.

References in corpus (7)

• Overcoming catastrophic forgetting in neural networks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Gradient Descent Happens in a Tiny Subspace
• On Warm-Starting Neural Network Training
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning