The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes

Alexander Atanasov, Blake Bordelon, Sabarish Sainathan +1

For small training set sizes $P$, the generalization error of wide neural networks is well-approximated by the error of an infinite width neural network (NN), either in the kernel…