Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation

Abstract

We investigate the stationary (late-time) training regime of single- and two-layer underparameterized linear neural networks within the continuum limit of stochastic gradient descent (SGD) for synthetic Gaussian data. In the case of a single-layer network in the weakly underparameterized regime, the spectrum of the noise covariance matrix deviates notably from the Hessian, which can be attributed to the broken detailed balance of SGD dynamics. The weight fluctuations are in this case generally anisotropic, but effectively experience an isotropic loss. For an underparameterized two-layer network, we describe the stochastic dynamics of the weights in each layer and analyze the associated stationary covariances. We identify the inter-layer coupling as a distinct source of anisotropy for the weight fluctuations. In contrast to the single-layer case, the weight fluctuations are effectively subject to an anisotropic loss, the flatness of which is inversely related to the fluctuation variance. We thereby provide an analytical derivation of the recently observed inverse variance-flatness relation in a model of a deep linear neural network.

arXiv comment27 pages, 9 figures

References in corpus (19)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Fluctuation-Dissipation: Response Theory in Statistical Physics
• The effective temperature
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Stochastic Dynamical Structure (SDS) of Nonequilibrium Processes in the Absence of Detailed Balance. II: construction of SDS with nonlinear force and multiplicative noise
• A mathematical theory of semantic development in deep neural networks
• The Principles of Deep Learning Theory
• Cleaning large correlation matrices: tools from random matrix theory
• Structure of Stochastic Dynamics near Fixed Points
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Stochastic Dynamical Structure (SDS) of Nonequilibrium Processes in the Absence of Detailed Balance. III: potential function in local stochastic dynamics and in steady state of Boltzmann-Gibbs type distribution function
• Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
• Stochastic gradient descent introduces an effective landscape-dependent regularization favoring flat solutions
• Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup
• Tracer particle in a confined correlated medium: an adiabatic elimination method
• Fluctuation-dissipation Type Theorem in Stochastic Linear Learning
• The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
• Stochastic Gradient Descent and Anomaly of Variance-flatness Relation in Artificial Neural Networks

Cited by in corpus (2)

• Noise Balance and Stationary Distribution of Stochastic Gradient Descent
• Anti-Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances in Flat Directions