Reinitializing weights vs units for maintaining plasticity in neural networks

J. Fernando Hernandez-Garcia, Shibhansh Dohare, Jun Luo +1

Loss of plasticity is a phenomenon in which a neural network loses its ability to learn when trained for an extended time on non-stationary data. It is a crucial problem to overcom…