Misalignment of Low-Loss Regions Causes Grokking

Yongding Tian, Zaid Al-Ars, Maksim Kitsak +1

Grokking refers to the delayed emergence of validation-set generalization after a model has already overfit the training set. Although first observed in small algorithmic tasks tra…