Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances

Abstract

We study how permutation symmetries in overparameterized multi-layer neural networks generate `symmetry-induced' critical points. Assuming a network with $L$ layers of minimal widths $r_1^{\ast },\dots ,r_{L-1}^{\ast }$ reaches a zero-loss minimum at $r_1^{\ast }!\cdots r_{L-1}^{\ast }!$ isolated points that are permutations of one another, we show that adding one extra neuron to each layer is sufficient to connect all these previously discrete minima into a single manifold. For a two-layer overparameterized network of width $r^{\ast }+h=:m$ we explicitly describe the manifold of global minima: it consists of $T(r^{\ast },m)$ affine subspaces of dimension at least $h$ that are connected to one another. For a network of width $m$, we identify the number $G(r,m)$ of affine subspaces containing only symmetry-induced critical points that are related to the critical points of a smaller network of width $r<r^{\ast }$. Via a combinatorial analysis, we derive closed-form formulas for $T$ and $G$ and show that the number of symmetry-induced critical subspaces dominates the number of affine subspaces forming the global minima manifold in the mildly overparameterized regime (small $h$) and vice versa in the vastly overparameterized regime ($h\gg r^{\ast }$). Our results provide new insights into the minimization of the non-convex loss function of overparameterized neural networks.

arXiv comment29 pages, 12 figures, ICML 2021

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• How to Escape Saddle Points Efficiently
• Essentially No Barriers in Neural Network Energy Landscape
• The statistics of critical points of Gaussian fields on large-dimensional spaces
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Gradient Descent Can Take Exponential Time to Escape Saddle Points
• Explorations on high dimensional landscapes
• Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
• Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
• On Connected Sublevel Sets in Deep Learning
• Noether: The More Things Change, the More Stay the Same
• The critical locus of overparameterized neural networks
• GENNI: Visualising the Geometry of Equivalences for Neural Network Identifiability

Cited by in corpus (5)

• Unification of Symmetries Inside Neural Networks: Transformer, Feedforward and Neural ODE
• Neuronal diversity can improve machine learning for physics and beyond
• Some observations on the ambivalent role of symmetries in Bayesian inference problems
• Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks