A Decentralized Approach to Bayesian Learning
arXiv:2007.06799
Abstract
Motivated by decentralized approaches to machine learning, we propose a collaborative Bayesian learning algorithm taking the form of decentralized Langevin dynamics in a non-convex setting. Our analysis show that the initial KL-divergence between the Markov Chain and the target posterior distribution is exponentially decreasing while the error contributions to the overall KL-divergence from the additive noise is decreasing in polynomial time. We further show that the polynomial-term experiences speed-up with number of agents and provide sufficient conditions on the time-varying step-sizes to guarantee convergence to the desired distribution. The performance of the proposed algorithm is evaluated on a wide variety of machine learning tasks. The empirical results show that the performance of individual agents with locally available data is on par with the centralized setting with considerable improvement in the convergence rate.
arXiv comment42 pages, 37 figures
References in corpus (20)
- Stochastic Gradient Hamiltonian Monte Carlo
- User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient
- Sampling Can Be Faster Than Optimization
- On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
- Parallelizing MCMC via Weierstrass Sampler
- Underdamped Langevin MCMC: A non-asymptotic analysis
- Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithm
- Convergence of Langevin MCMC in KL-divergence
- Parallelizing MCMC with Random Partition Trees
- High-Order Langevin Diffusion Yields an Accelerated MCMC Algorithm
- Is There an Analog of Nesterov Acceleration for MCMC?
- Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Suffices
- On stochastic gradient Langevin dynamics with dependent data streams: the fully non-convex case
- Dimensionally Tight Bounds for Second-Order Hamiltonian Monte Carlo
- Decentralized Stochastic Gradient Langevin Dynamics and Hamiltonian Monte Carlo
- Nonasymptotic estimates for Stochastic Gradient Langevin Dynamics under local conditions in nonconvex optimization
- Variational consensus Monte Carlo
- SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Stochastic Optimization
- Decentralized Langevin Dynamics
- Computational Separations between Sampling and Optimization