Feature-Learning Networks Are Consistent Across Widths At Realistic Scales

Nikhil Vyas, Alexander Atanasov, Blake Bordelon +3

We study the effect of width on the dynamics of feature-learning neural networks across a variety of architectures and datasets. Early in training, wide neural networks trained on…