Laplacian Smoothing Stochastic Gradient Markov Chain Monte Carlo

Abstract

As an important Markov Chain Monte Carlo (MCMC) method, stochastic gradient Langevin dynamics (SGLD) algorithm has achieved great success in Bayesian learning and posterior sampling. However, SGLD typically suffers from slow convergence rate due to its large variance caused by the stochastic gradient. In order to alleviate these drawbacks, we leverage the recently developed Laplacian Smoothing (LS) technique and propose a Laplacian smoothing stochastic gradient Langevin dynamics (LS-SGLD) algorithm. We prove that for sampling from both log-concave and non-log-concave densities, LS-SGLD achieves strictly smaller discretization error in $2$-Wasserstein distance, although its mixing rate can be slightly slower. Experiments on both synthetic and real datasets verify our theoretical results, and demonstrate the superior performance of LS-SGLD on different machine learning tasks including posterior sampling, Bayesian logistic regression and training Bayesian convolutional neural networks. The code is available at \url{https://github.com/BaoWangMath/LS-MCMC}.

arXiv comment27 pages, 5 figures

References in corpus (9)

• Stochastic Gradient Hamiltonian Monte Carlo
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
• Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithm
• Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints
• Laplacian Smoothing Gradient Descent
• On the Theory of Variance Reduction for Stochastic Gradient Monte Carlo
• DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERM
• Stochastic Gradient Hamiltonian Monte Carlo with Variance Reduction for Bayesian Inference

Cited by in corpus (1)

• Faster Convergence of Stochastic Gradient Langevin Dynamics for Non-Log-Concave Sampling