Stochastic Gradient Descent and Anomaly of Variance-flatness Relation in Artificial Neural Networks

Abstract

Stochastic gradient descent (SGD), a widely used algorithm in deep-learning neural networks has attracted continuing studies for the theoretical principles behind its success. A recent work reports an anomaly (inverse) relation between the variance of neural weights and the landscape flatness of the loss function driven under SGD [Feng & Tu, PNAS 118, 0027 (2021)]. To investigate this seemingly violation of statistical physics principle, the properties of SGD near fixed points are analysed via a dynamic decomposition method. Our approach recovers the true "energy" function under which the universal Boltzmann distribution holds. It differs from the cost function in general and resolves the paradox raised by the the anomaly. The study bridges the gap between the classical statistical mechanics and the emerging discipline of artificial intelligence, with potential for better algorithms to the latter.

References in corpus (12)

• Overcoming catastrophic forgetting in neural networks
• Machine learning and the physical sciences
• A high-bias, low-variance introduction to Machine Learning for physicists
• Stochastic Dynamical Structure (SDS) of Nonequilibrium Processes in the Absence of Detailed Balance. II: construction of SDS with nonlinear force and multiplicative noise
• Structure of Stochastic Dynamics near Fixed Points
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• The statistics of critical points of Gaussian fields on large-dimensional spaces
• Shaping the learning landscape in neural networks around wide flat minima
• Statistical mechanics of complex neural systems and high dimensional data
• Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
• Fluctuation-dissipation Type Theorem in Stochastic Linear Learning
• Topology, Vorticity and Limit Cycle in a Stabilized Kuramoto-Sivashinsky Equation

Cited by in corpus (1)

• Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation