Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry

Abstract

We consider the optimization problem associated with fitting two-layers ReLU networks with respect to the squared loss, where labels are generated by a target network. We leverage the rich symmetry structure to analytically characterize the Hessian at various families of spurious minima in the natural regime where the number of inputs $d$ and the number of hidden neurons $k$ is finite. In particular, we prove that for $d\ge k$ standard Gaussian inputs: (a) of the $dk$ eigenvalues of the Hessian, $dk-O(d)$ concentrate near zero, (b) $\Omega (d)$ of the eigenvalues grow linearly with $k$. Although this phenomenon of extremely skewed spectrum has been observed many times before, to our knowledge, this is the first time it has been established {rigorously}. Our analytic approach uses techniques, new to the field, from symmetry breaking and representation theory, and carries important implications for our ability to argue about statistical generalization through local curvature.

References in corpus (16)

• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Three Factors Influencing Minima in SGD
• Qualitatively characterizing neural network optimization problems
• Essentially No Barriers in Neural Network Energy Landscape
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• Sharp Minima Can Generalize For Deep Nets
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
• Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Porcupine Neural Networks: (Almost) All Local Optima are Global
• On the Principle of Least Symmetry Breaking in Shallow ReLU Models

Cited by in corpus (4)

• Symmetry & critical points for a model shallow neural network
• Symmetry Breaking in Symmetric Tensor Decomposition
• The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks
• Equivariant bifurcation, quadratic equivariants, and symmetry breaking for the standard representation of $S_n$