Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias

Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey +2

Randomly initialized neural networks induce a prior over functions, but the predictor used in practice is produced only after training. We ask how much of this initial bias survive…