Showing 2024Show all
3 papers · 1 filter
cs.LG2024
Plastic Learning with Deep Fourier Features
Alex Lewandowski, Dale Schuurmans, Marlos C. Machado
Deep neural networks can struggle to learn continually in the face of non-stationarity. This phenomenon is known as loss of plasticity. In this paper, we identify underlying princi…
cs.LG2024
Toward Understanding In-context vs. In-weight Learning
Bryan Chan, Xinyi Chen, András György +1
It has recently been demonstrated empirically that in-context learning emerges in transformers when certain distributional properties are present in the training data, but this abi…
cs.LG2024
Learning Continually by Spectral Regularization
Alex Lewandowski, Michał Bortkiewicz, Saurabh Kumar +4
Loss of plasticity is a phenomenon where neural networks can become more difficult to train over the course of learning. Continual learning algorithms seek to mitigate this effect…