Stochastic Gradient and Langevin Processes

Abstract

We prove quantitative convergence rates at which discrete Langevin-like processes converge to the invariant distribution of a related stochastic differential equation. We study the setup where the additive noise can be non-Gaussian and state-dependent and the potential function can be non-convex. We show that the key properties of these processes depend on the potential function and the second moment of the additive noise. We apply our theoretical findings to studying the convergence of Stochastic Gradient Descent (SGD) for non-convex problems and corroborate them with experiments using SGD to train deep neural networks on the CIFAR-10 dataset.

arXiv commentICML 2020, code available at https://github.com/dongyin92/noise_covariance

Cited by in corpus (7)

• Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
• The Heavy-Tail Phenomenon in SGD
• On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
• Online stochastic gradient descent on non-convex losses from high-dimensional inference
• Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate
• Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework