Extreme Memorization via Scale of Initialization

Harsh Mehta, Ashok Cutkosky, Behnam Neyshabur

We construct an experimental setup in which changing the scale of initialization strongly impacts the implicit regularization induced by SGD, interpolating from good generalization…