The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization

Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1

Grokking -- the sudden generalisation that appears long after a model has perfectly memorised its training data -- has been widely observed but lacks a quantitative theory explaini…