Stochastic Gradient Descent as Approximate Bayesian Inference

Abstract

Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a Markov chain with a stationary distribution. With this perspective, we derive several new results. (1) We show that constant SGD can be used as an approximate Bayesian posterior inference algorithm. Specifically, we show how to adjust the tuning parameters of constant SGD to best match the stationary distribution to a posterior, minimizing the Kullback-Leibler divergence between these two distributions. (2) We demonstrate that constant SGD gives rise to a new variational EM algorithm that optimizes hyperparameters in complex probabilistic models. (3) We also propose SGD with momentum for sampling and show how to adjust the damping coefficient accordingly. (4) We analyze MCMC algorithms. For Langevin Dynamics and Stochastic Gradient Fisher Scoring, we quantify the approximation errors due to finite learning rates. Finally (5), we use the stochastic process perspective to give a short proof of why Polyak averaging is optimal. Based on this idea, we propose a scalable approximate MCMC algorithm, the Averaged Stochastic Gradient Sampler.

arXiv comment35 pages, published version (JMLR 2017)

References in corpus (11)

• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• Stochastic Gradient Hamiltonian Monte Carlo
• A Complete Recipe for Stochastic Gradient MCMC
• On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
• Automatic Variational Inference in Stan
• From Averaging to Acceleration, There is Only a Step-size
• Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization
• Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chains
• The Generalized Reparameterization Gradient

Cited by in corpus (29)

• Don't Decay the Learning Rate, Increase the Batch Size
• Deep Neural Networks as Gaussian Processes
• Regularizing and Optimizing LSTM Language Models
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Vertex and Energy Reconstruction in JUNO with Machine Learning Methods
• Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chains
• HiGrad: Uncertainty Quantification for Online Learning and Stochastic Approximation
• On the Validity of Bayesian Neural Networks for Uncertainty Estimation
• Classification and image processing with a semi-discrete scheme for fidelity forced Allen--Cahn on graphs
• Repulsive Deep Ensembles are Bayesian
• Vprop: Variational Inference using RMSprop
• Scalable Bayesian Approach for the DINA Q-matrix Estimation Combining Stochastic Optimization and Variational Inference
• A Gaussian Process perspective on Convolutional Neural Networks
• A Modularized Efficient Framework for Non-Markov Time Series Estimation
• Deep Bayesian ICP Covariance Estimation
• Scalable Natural Gradient Langevin Dynamics in Practice
• Data Augmentation for Bayesian Deep Learning
• On Accelerating Distributed Convex Optimizations
• On the Distributional Properties of Adaptive Gradients
• First-Passage Approach to Optimizing Perturbations for Improved Training of Machine Learning Models
• A Bayesian neural network predicts the dissolution of compact planetary systems
• Introspective Generative Modeling: Decide Discriminatively
• Variational Encoders and Autoencoders : Information-theoretic Inference and Closed-form Solutions
• Extending Label Smoothing Regularization with Self-Knowledge Distillation
• Generalisation in fully-connected neural networks for time series forecasting
• Enhance Diffusion to Improve Robust Generalization
• Mixing between the Cross Entropy and the Expectation Loss Terms
• Batch size-invariance for policy optimization
• Learnability Window in Gated Recurrent Neural Networks