Learning through atypical "phase transitions" in overparameterized neural networks
arXiv:2110.00683 · doi:10.1103/PhysRevE.106.014116
Abstract
Current deep neural networks are highly overparameterized (up to billions of connection weights) and nonlinear. Yet they can fit data almost perfectly through variants of gradient descent algorithms and achieve unexpected levels of prediction accuracy without overfitting. These are formidable results that defy predictions of statistical learning and pose conceptual challenges for non-convex optimization. In this paper, we use methods from statistical physics of disordered systems to analytically study the computational fallout of overparameterization in non-convex binary neural network models, trained on data generated from a structurally simpler but "hidden" network. As the number of connection weights increases, we follow the changes of the geometrical structure of different minima of the error loss function and relate them to learning and generalization performance. A first transition happens at the so-called interpolation point, when solutions begin to exist (perfect fitting becomes possible). This transition reflects the properties of typical solutions, which however are in sharp minima and hard to sample. After a gap, a second transition occurs, with the discontinuous appearance of a different kind of "atypical" structures: wide regions of the weight space that are particularly solution-dense and have good generalization properties. The two kinds of solutions coexist, with the typical ones being exponentially more numerous, but empirically we find that efficient algorithms sample the atypical, rare ones. This suggests that the atypical phase transition is the relevant one for learning. The results of numerical tests with realistic networks on observables suggested by the theory are consistent with this scenario.
arXiv comment28 pages, 14 figures
References in corpus (24)
- Many body localization and thermalization in quantum statistical mechanics
- Reconciling modern machine learning practice and the bias-variance trade-off
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Binarized Neural Networks
- Deep Neural Networks as Gaussian Processes
- Recipes for metastable states in Spin Glasses
- Scaling description of generalization with number of parameters in deep learning
- Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
- Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
- Efficient supervised learning in networks with binary synapses
- Modelling the influence of data structure on learning in neural networks: the hidden manifold model
- Shaping the learning landscape in neural networks around wide flat minima
- Generalisation error in learning with random features and the hidden manifold model
- A jamming transition from under- to over-parametrization affects loss landscape and generalization
- Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
- Origin of the computational hardness for learning with binary synapses
- Efficiency of quantum versus classical annealing in non-convex learning problems
- Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
- Unveiling the structure of wide flat minima in neural networks
- Entropic gradient descent algorithms and wide flat minima
- Generalization learning in a perceptron with binary synapses
- Clustering of solutions in the symmetric binary perceptron
- Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures
Cited by in corpus (9)
- A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
- Phase transitions in the mini-batch size for sparse and dense two-layer neural networks
- Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networks
- Eight challenges in developing theory of intelligence
- Daydreaming Hopfield Networks and their surprising effectiveness on correlated data
- Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation
- Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problems
- Supervised Learning Guarantee for Quantum AdaBoost
- Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings