Shaping the learning landscape in neural networks around wide flat minima

Abstract

Learning in Deep Neural Networks (DNN) takes place by minimizing a non-convex high-dimensional loss function, typically by a stochastic gradient descent (SGD) strategy. The learning process is observed to be able to find good minimizers without getting stuck in local critical points, and that such minimizers are often satisfactory at avoiding overfitting. How these two features can be kept under control in nonlinear devices composed of millions of tunable connections is a profound and far reaching open question. In this paper we study basic non-convex one- and two-layer neural network models which learn random patterns, and derive a number of basic geometrical and algorithmic features which suggest some answers. We first show that the error loss function presents few extremely wide flat minima (WFM) which coexist with narrower minima and critical points. We then show that the minimizers of the cross-entropy loss function overlap with the WFM of the error loss. We also show examples of learning devices for which WFM do not exist. From the algorithmic perspective we derive entropy driven greedy and message passing algorithms which focus their search on wide flat regions of minimizers. In the case of SGD and cross-entropy loss, we show that a slow reduction of the norm of the weights along the learning process also leads to WFM. We corroborate the results by a numerical study of the correlations between the volumes of the minimizers, their Hessian and their generalization performance on real data.

arXiv comment37 pages (16 main text), 10 figures (7 main text)

References in corpus (14)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• The structural glass transition and the entropy of the metastable states
• Recipes for metastable states in Spin Glasses
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Learning by message-passing in networks of discrete synapses
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• Efficient supervised learning in networks with binary synapses
• Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Network
• Origin of the computational hardness for learning with binary synapses
• Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
• Local entropy as a measure for sampling solutions in Constraint Satisfaction Problems
• Generalization learning in a perceptron with binary synapses
• Learning may need only a few bits of synaptic precision
• On the role of synaptic stochasticity in training low-precision neural networks

Cited by in corpus (34)

• Neural network wave functions and the sign problem
• The Global Landscape of Neural Networks: An Overview
• Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
• Unveiling the structure of wide flat minima in neural networks
• Learning through atypical "phase transitions" in overparameterized neural networks
• Entropic gradient descent algorithms and wide flat minima
• Model-Based Policy Search Using Monte Carlo Gradient Estimation with Real Systems Application
• On the different regimes of Stochastic Gradient Descent
• Spin glass theory and its new challenge: structured disorder
• Data efficiency and extrapolation trends in neural network interatomic potentials
• Replica symmetry breaking in dense neural networks
• Clustering of solutions in the symmetric binary perceptron
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• Deep learning via message passing algorithms based on belief propagation
• Generalized hetero-associative neural networks
• Visualizing high-dimensional loss landscapes with Hessian directions
• Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures
• Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networks
• A spin-glass model for the loss surfaces of generative adversarial networks
• Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
• When you can't count, sample! Computable entropies beyond equilibrium from basin volumes
• Analytic Insights into Structure and Rank of Neural Network Hessian Maps
• Data-driven effective model shows a liquid-like deep learning
• Recognition Capabilities of a Hopfield Model with Auxiliary Hidden Neurons
• FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustness
• Layer-wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learning
• Stochastic Gradient Descent and Anomaly of Variance-flatness Relation in Artificial Neural Networks
• Rigorous approaches for spin glass and Gaussian spin glass with P-wise interactions
• Native state of natural proteins optimises local entropy
• Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry
• Training Multi-Layer Binary Neural Networks With Local Binary Error Signals
• Architecture-Aware Minimization (A${}^2$M): How to Find Flat Minima in Neural Architecture Search
• Detecting underdetermination in parameterized quantum circuits
• The maximum-average subtensor problem: equilibrium and out-of-equilibrium properties