Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances

Berfin Şimşek, François Ged, Arthur Jacot +4

We study how permutation symmetries in overparameterized multi-layer neural networks generate `symmetry-induced' critical points. Assuming a network with $L$ layers of minimal wi…