Tunneling the Loss Landscape: Bypassing Memorization with Monte Carlo Parameter Swapping

Lai Shun Chan, Xiaotian Zhang, Yue Shang +2

Grokking is a striking phenomenon in neural network training, where a model can undergo a prolonged period of pure memorization before abrupt generalization. While previous works h…