A Variational Analysis of Stochastic Gradient Algorithms

Abstract

Stochastic Gradient Descent (SGD) is an important algorithm in machine learning. With constant learning rates, it is a stochastic process that, after an initial phase of convergence, generates samples from a stationary distribution. We show that SGD with constant rates can be effectively used as an approximate posterior inference algorithm for probabilistic modeling. Specifically, we show how to adjust the tuning parameters of SGD such as to match the resulting stationary distribution to the posterior. This analysis rests on interpreting SGD as a continuous-time stochastic process and then minimizing the Kullback-Leibler divergence between its stationary distribution and the target posterior. (This is in the spirit of variational inference.) In more detail, we model SGD as a multivariate Ornstein-Uhlenbeck process and then use properties of this process to derive the optimal parameters. This theoretical framework also connects SGD to modern scalable inference algorithms; we analyze the recently proposed stochastic gradient Fisher scoring under this perspective. We demonstrate that SGD with properly chosen constant rates gives a new way to optimize hyperparameters in probabilistic models.

arXiv comment8 pages, 3 figures

References in corpus (7)

• Stochastic Gradient Hamiltonian Monte Carlo
• A Complete Recipe for Stochastic Gradient MCMC
• On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
• Automatic Variational Inference in Stan
• From Averaging to Acceleration, There is Only a Step-size
• Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization
• Early Stopping is Nonparametric Variational Inference

Cited by in corpus (17)

• Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
• Analysis of Stochastic Gradient Descent in Continuous Time
• Multiplicative noise and heavy tails in stochastic optimization
• Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chains
• Learning to Draw Samples with Amortized Stein Variational Gradient Descent
• Scalable Generalized Dynamic Topic Models
• Approximate Inference with Amortised MCMC
• Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks
• Two Methods For Wild Variational Inference
• Quasi-potential as an implicit regularizer for the loss function in the stochastic gradient descent
• Online Learning in Contextual Bandits using Gated Linear Networks
• Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms
• On generalization bounds for deep networks based on loss surface implicit regularization
• Differential Equations for Modeling Asynchronous Algorithms
• Distribution-Dependent Analysis of Gibbs-ERM Principle