The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects

Abstract

Understanding the behavior of stochastic gradient descent (SGD) in the context of deep neural networks has raised lots of concerns recently. Along this line, we study a general form of gradient based optimization dynamics with unbiased noise, which unifies SGD and standard Langevin dynamics. Through investigating this general optimization dynamics, we analyze the behavior of SGD on escaping from minima and its regularization effects. A novel indicator is derived to characterize the efficiency of escaping from minima through measuring the alignment of noise covariance and the curvature of loss function. Based on this indicator, two conditions are established to show which type of noise structure is superior to isotropic noise in term of escaping efficiency. We further show that the anisotropic noise in SGD satisfies the two conditions, and thus helps to escape from sharp and poor minima effectively, towards more stable and flat minima that typically generalize well. We systematically design various experiments to verify the benefits of the anisotropic noise, compared with full gradient descent plus isotropic diffusion (i.e. Langevin dynamics).

arXiv commentICML 2019 camera ready

Cited by in corpus (52)

• Don't Use Large Mini-Batches, Use Local SGD
• Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• The Heavy-Tail Phenomenon in SGD
• Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
• The Implicit Regularization of Stochastic Gradient Flow for Least Squares
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• On the different regimes of Stochastic Gradient Descent
• The Implicit and Explicit Regularization Effects of Dropout
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noise
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
• Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
• On Training Implicit Models
• A Study of Gradient Variance in Deep Learning
• First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise
• How noise affects the Hessian spectrum in overparameterized neural networks
• Drive-specific adaptation in disordered mechanical networks of bistable springs
• On the fast convergence of random perturbations of the gradient flow
• Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization
• Dynamic of Stochastic Gradient Descent with State-Dependent Noise
• Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms
• On the interplay between noise and curvature and its effect on optimization and generalization
• Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting
• On stochastic mirror descent with interacting particles: convergence properties and variance reduction
• Stochastic Training is Not Necessary for Generalization
• S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima
• Partial local entropy and anisotropy in deep weight spaces
• SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
• Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics
• Never Go Full Batch (in Stochastic Convex Optimization)
• On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Control
• Isotropic SGD: a Practical Approach to Bayesian Posterior Sampling
• Escaping Saddle Points with Stochastically Controlled Stochastic Gradient Methods
• A Gradient Estimator for Time-Varying Electrical Networks with Non-Linear Dissipation
• Contrastive Weight Regularization for Large Minibatch SGD
• Noise-induced degeneration in online learning
• Training Efficiency and Robustness in Deep Learning
• Adaptive Weight Decay for Deep Neural Networks
• Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks
• Mixing between the Cross Entropy and the Expectation Loss Terms
• Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD
• Inherent Noise in Gradient Based Methods
• Analytic expressions for the output evolution of a deep neural network
• Deep Learning Theory through the Lens of Statistical Physics: A Topical Review