Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

Shuang Liang, Tom Jacobs, Guido Montúfar

We study the implicit bias of noisy stochastic gradient descent in training wide two-layer ReLU networks for multivariate regression. In a mean-field regime, the training dynamics…