Grokking as Structural Inference: Transformers Need Bayesian Lottery Tickets

Kai Hidajat, Solden Stoll, Joseph An

Why does a Transformer that has memorized its training set wait thousands of steps before it generalizes? Existing accounts locate this delay in norm minimization, feature emergenc…