On the symmetries in the dynamics of wide two-layer neural networks

Karl Hajjar, Lenaic Chizat

We consider the idealized setting of gradient flow on the population risk for infinitely wide two-layer ReLU neural networks (without bias), and study the effect of symmetries on t…