Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning

Abstract

Finding parameters that minimise a loss function is at the core of many machine learning methods. The Stochastic Gradient Descent algorithm is widely used and delivers state of the art results for many problems. Nonetheless, Stochastic Gradient Descent typically cannot find the global minimum, thus its empirical effectiveness is hitherto mysterious. We derive a correspondence between parameter inference and free energy minimisation in statistical physics. The degree of undersampling plays the role of temperature. Analogous to the energy-entropy competition in statistical physics, wide but shallow minima can be optimal if the system is undersampled, as is typical in many applications. Moreover, we show that the stochasticity in the algorithm has a non-trivial correlation structure which systematically biases it towards wide minima. We illustrate our argument with two prototypical models: image classification using deep learning, and a linear neural network where we can analytically reveal the relationship between entropy and out-of-sample error.

References in corpus (11)

• Artificial Intelligence and Statistics
• Three Factors Influencing Minima in SGD
• The sloppy model universality class and the Vandermonde matrix
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• The statistics of critical points of Gaussian fields on large-dimensional spaces
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Perspective: Energy Landscapes for Machine Learning
• Statistical mechanics of complex neural systems and high dimensional data
• Depth Creates No Bad Local Minima
• On the diffusion approximation of nonconvex stochastic gradient descent
• Structural analysis of high-dimensional basins of attraction

Cited by in corpus (25)

• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Dimensionality compression and expansion in Deep Neural Networks
• Non-classical nucleation pathways in stacking-disordered crystals
• Stochastic gradient descent introduces an effective landscape-dependent regularization favoring flat solutions
• On the different regimes of Stochastic Gradient Descent
• Geometry of energy landscapes and the optimizability of deep neural networks
• Information Complexity and Generalization Bounds
• Generalization bounds for deep learning
• Efficient ConvNets for Analog Arrays
• How noise affects the Hessian spectrum in overparameterized neural networks
• Deep Learning is Singular, and That's Good
• A continuous-time analysis of distributed stochastic gradient
• Dynamic of Stochastic Gradient Descent with State-Dependent Noise
• Is SGD a Bayesian sampler? Well, almost
• How neural networks find generalizable solutions: Self-tuned annealing in deep learning
• Thermodynamics of the Ising model encoded in restricted Boltzmann machines
• Why flatness does and does not correlate with generalization for deep neural networks
• Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation
• Stochastic Gradient Descent and Anomaly of Variance-flatness Relation in Artificial Neural Networks
• Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent
• Training Deep Neural Networks by optimizing over nonlocal paths in hyperparameter space
• Phases of learning dynamics in artificial neural networks: with or without mislabeled data
• Anti-Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances in Flat Directions
• MGA: Momentum Gradient Attack on Network
• Parallel Stochastic Optimization Framework for Large-Scale Non-Convex Stochastic Problems