A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation

Etienne Boursier, Scott Pesme, Radu-Alexandru Dragomir

We study the dynamics of gradient flow with small weight decay on general training losses $F:{\mathbb{R}}^d\to \mathbb{R}$. Under mild regularity assumptions and assuming convergen…