4 papers
What Can Grokking Teach Us About Learning Under Nonstationarity?
Clare Lyle, Gharda Sokar, Razvan Pascanu +1
In continual learning problems, it is often necessary to overwrite components of a neural network's learned representation in response to changes in the data stream; however, neura…
Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset
Alexandre Galashov, Michalis K. Titsias, András György +4
Neural networks are traditionally trained under the assumption that data come from a stationary distribution. However, settings which violate this assumption are becoming more popu…
Learning Continually by Spectral Regularization
Alex Lewandowski, MichaÅ Bortkiewicz, Saurabh Kumar +4
Loss of plasticity is a phenomenon where neural networks can become more difficult to train over the course of learning. Continual learning algorithms seek to mitigate this effect…
Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models
Amal Rannen-Triki, Jorg Bornschein, Razvan Pascanu +5
We consider the problem of online fine tuning the parameters of a language model at test time, also known as dynamic evaluation. While it is generally known that this approach impr…