Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape

Abstract

The permutation symmetry of neurons in each layer of a deep neural network gives rise not only to multiple equivalent global minima of the loss function, but also to first-order saddle points located on the path between the global minima. In a network of $d-1$ hidden layers with $n_k$ neurons in layers $k=1,\dots ,d$, we construct smooth paths between equivalent global minima that lead through a `permutation point' where the input and output weight vectors of two neurons in the same hidden layer $k$ collide and interchange. We show that such permutation points are critical points with at least $n_{k+1}$ vanishing eigenvalues of the Hessian matrix of second derivatives indicating a local plateau of the loss function. We find that a permutation point for the exchange of neurons $i$ and $j$ transits into a flat valley (or generally, an extended plateau of $n_{k+1}$ flat dimensions) that enables all $n_k!$ permutations of neurons in a given layer $k$ at the same loss value. Moreover, we introduce high-order permutation points by exploiting the recursive structure in neural network functions, and find that the number of $K^{\text{th}}$-order permutation points is at least by a factor ${\sum }_{k=1}^{d-1}\frac{1}{2{!}^K}\left(\genfrac{}{}{0ex}{}{n_k-K}{K}\right)$ larger than the (already huge) number of equivalent global minima. In two tasks, we illustrate numerically that some of the permutation points correspond to first-order saddles (`permutation saddles'): first, in a toy network with a single hidden layer on a function approximation task and, second, in a multilayer network on the MNIST task. Our geometric approach yields a lower bound on the number of critical points generated by weight-space symmetries and provides a simple intuitive link between previous mathematical results and numerical observations.

References in corpus (16)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding deep learning requires rethinking generalization
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Qualitatively characterizing neural network optimization problems
• Essentially No Barriers in Neural Network Energy Landscape
• The loss surface of deep and wide neural networks
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Gradient Descent Converges to Minimizers
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• Perspective: Energy Landscapes for Machine Learning
• Depth Creates No Bad Local Minima
• Explorations on high dimensional landscapes

Cited by in corpus (7)

• Beyond Convexity -- Contraction and Global Convergence of Gradient Descent
• Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
• Unification of Symmetries Inside Neural Networks: Transformer, Feedforward and Neural ODE
• Optimizing Mode Connectivity via Neuron Alignment
• On the Principle of Least Symmetry Breaking in Shallow ReLU Models
• GENNI: Visualising the Geometry of Equivalences for Neural Network Identifiability
• When Are Solutions Connected in Deep Networks?