Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networks

Ibrahim Talha Ersoy, Karoline Wiesner

Deep neural networks (DNNs) exhibit first order phase transitions under variations of the L2 regularization strength, with each transition marking the onset of a new learnable feat…