A Basin-Selection Perspective on Grokking via Singular Learning Theory

Ben Cullen, Sergio Estan-Ruiz, Riya Danait +1

Grokking, the abrupt transition from memorization to generalisation after extended training, suggests the presence of competing solution basins with distinct statistical properties…