A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions
arXiv:2010.09800
Abstract
We propose an adaptively weighted stochastic gradient Langevin dynamics algorithm (SGLD), so-called contour stochastic gradient Langevin dynamics (CSGLD), for Bayesian learning in big data statistics. The proposed algorithm is essentially a \emph{scalable dynamic importance sampler}, which automatically \emph{flattens} the target distribution such that the simulation for a multi-modal distribution can be greatly facilitated. Theoretically, we prove a stability condition and establish the asymptotic convergence of the self-adapting parameter to a {\it unique fixed-point}, regardless of the non-convexity of the original energy function; we also present an error analysis for the weighted averaging estimators. Empirically, the CSGLD algorithm is tested on multiple benchmark datasets including CIFAR10 and CIFAR100. The numerical results indicate its superiority to avoid the local trap problem in training deep neural networks.
arXiv commentAccepted by NeurIPS 2020
References in corpus (17)
- An efficient, multiple range random walk algorithm to calculate the density of states
- Random Erasing Data Augmentation
- Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
- Stochastic Gradient Hamiltonian Monte Carlo
- On the ergodicity properties of some adaptive MCMC algorithms
- A Complete Recipe for Stochastic Gradient MCMC
- Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
- On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
- Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
- Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring
- Convergence of adaptive and interacting Markov chain Monte Carlo algorithms
- Bayesian GAN
- Stochastic Runge-Kutta Accelerates Langevin Monte Carlo and Beyond
- Non-convex Learning via Replica Exchange Stochastic Gradient MCMC
- Interacting Contour Stochastic Gradient Langevin Dynamics
- Accelerating Convergence of Replica Exchange Stochastic Gradient MCMC via Variance Reduction
- Convex Optimization with Unbounded Nonconvex Oracles using Simulated Annealing