A Spectral Theory of Grokking: Weight Decay induces Feature Learning

Lenz Pracher, Pascal de Jong, Oskar Lieshaus +2

In grokking an early fit to the training data separates from a much later improvement in generalization. During this delay, training can move from a fixed neural tangent kernel (NT…