Preserving Plasticity in Continual Learning via Dynamical Isometry

Andries Rosseau, Robert Müller, Ann Nowé

Continual training of deep neural networks under non-stationarity often leads to a progressive loss of plasticity, eventually limiting further learning. We relate plasticity to the…