Unveiling the structure of wide flat minima in neural networks

Abstract

The success of deep learning has revealed the application potential of neural networks across the sciences and opened up fundamental theoretical problems. In particular, the fact that learning algorithms based on simple variants of gradient methods are able to find near-optimal minima of highly nonconvex loss functions is an unexpected feature of neural networks. Moreover, such algorithms are able to fit the data even in the presence of noise, and yet they have excellent predictive capabilities. Several empirical results have shown a reproducible correlation between the so-called flatness of the minima achieved by the algorithms and the generalization performance. At the same time, statistical physics results have shown that in nonconvex networks a multitude of narrow minima may coexist with a much smaller number of wide flat minima, which generalize well. Here we show that wide flat minima arise as complex extensive structures, from the coalescence of minima around "high-margin" (i.e., locally robust) configurations. Despite being exponentially rare compared to zero-margin ones, high-margin minima tend to concentrate in particular regions. These minima are in turn surrounded by other solutions of smaller and smaller margin, leading to dense regions of solutions over long distances. Our analysis also provides an alternative analytical method for estimating when flat minima appear and when algorithms begin to find solutions, as the number of model parameters varies.

arXiv comment15 pages, 8 figures

References in corpus (13)

• A high-bias, low-variance introduction to Machine Learning for physicists
• Recipes for metastable states in Spin Glasses
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Learning by message-passing in networks of discrete synapses
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• Efficient supervised learning in networks with binary synapses
• Shaping the learning landscape in neural networks around wide flat minima
• Origin of the computational hardness for learning with binary synapses
• Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
• Local entropy as a measure for sampling solutions in Constraint Satisfaction Problems
• Activation function dependence of the storage capacity of treelike neural networks
• Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures
• Data-driven effective model shows a liquid-like deep learning

Cited by in corpus (11)

• Learning through atypical "phase transitions" in overparameterized neural networks
• Arrangement of nearby minima and saddles in the mixed spherical energy landscapes
• Phase transitions in the mini-batch size for sparse and dense two-layer neural networks
• A spin-glass model for the loss surfaces of generative adversarial networks
• Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networks
• On the Atypical Solutions of the Symmetric Binary Perceptron
• Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problems
• Performance of machine-learning-assisted Monte Carlo in sampling from simple statistical physics models
• Biased thermodynamics can explain the behaviour of smart optimization algorithms that work above the dynamical threshold
• Algorithmic thresholds in combinatorial optimization depend on the time scaling
• Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry