Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts
arXiv:2009.09535
Abstract
Bayesian deep learning offers a principled way to address many issues concerning safety of artificial intelligence (AI), such as model uncertainty,model interpretability, and prediction bias. However, due to the lack of efficient Monte Carlo algorithms for sampling from the posterior of deep neural networks (DNNs), Bayesian deep learning has not yet powered our AI system. We propose a class of adaptive stochastic gradient Markov chain Monte Carlo (SGMCMC) algorithms, where the drift function is biased to enhance escape from saddle points and the bias is adaptively adjusted according to the gradient of past samples. We establish the convergence of the proposed algorithms under mild conditions, and demonstrate via numerical examples that the proposed algorithms can significantly outperform the existing SGMCMC algorithms, such as stochastic gradient Langevin dynamics (SGLD), stochastic gradient Hamiltonian Monte Carlo (SGHMC) and preconditioned SGLD, in both simulation and optimization tasks.
arXiv comment27 pages
References in corpus (12)
- ADADELTA: An Adaptive Learning Rate Method
- An overview of gradient descent optimization algorithms
- What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
- User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient
- A Complete Recipe for Stochastic Gradient MCMC
- Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
- Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
- On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
- Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
- The True Cost of Stochastic Gradient Langevin Dynamics
- Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations
- Extended Stochastic Gradient MCMC for Large-Scale Bayesian Variable Selection