The loss landscape of overparameterized neural networks

Abstract

We explore some mathematical features of the loss landscape of overparameterized neural networks. A priori one might imagine that the loss function looks like a typical function from ${\mathbb{R}}^n$ to $\mathbb{R}$ - in particular, nonconvex, with discrete global minima. In this paper, we prove that in at least one important way, the loss function of an overparameterized neural network does not look like a typical function. If a neural net has $n$ parameters and is trained on $d$ data points, with $n>d$, we show that the locus $M$ of global minima of $L$ is usually not discrete, but rather an $n-d$ dimensional submanifold of ${\mathbb{R}}^n$. In practice, neural nets commonly have orders of magnitude more parameters than data points, so this observation implies that $M$ is typically a very high-dimensional subset of ${\mathbb{R}}^n$.

arXiv comment9 pages

References in corpus (3)

• Understanding deep learning requires rethinking generalization
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys

Cited by in corpus (19)

• Scaling description of generalization with number of parameters in deep learning
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• On exponential convergence of SGD in non-convex over-parametrized learning
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• Beyond Convexity -- Contraction and Global Convergence of Gradient Descent
• Variational Quantum Classifiers Through the Lens of the Hessian
• Asymmetric Valleys: Beyond Sharp and Flat Local Minima
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Loss landscapes and optimization in over-parameterized non-linear systems and neural networks
• Taxonomizing local versus global structure in neural network loss landscapes
• Hierarchical Weight Averaging for Deep Neural Networks
• Intrinsically Sparse Long Short-Term Memory Networks
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• The critical locus of overparameterized neural networks
• Understanding over-parameterized deep networks by geometrization
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• On minimal representations of shallow ReLU networks
• Implicit regularization and solution uniqueness in over-parameterized matrix sensing