What Can Grokking Teach Us About Learning Under Nonstationarity?

Clare Lyle, Gharda Sokar, Razvan Pascanu +1

In continual learning problems, it is often necessary to overwrite components of a neural network's learned representation in response to changes in the data stream; however, neura…