QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning

Abstract

The objective of Federated Learning (FL) is to perform statistical inference for data which are decentralised and stored locally on networked clients. FL raises many constraints which include privacy and data ownership, communication overhead, statistical heterogeneity, and partial client participation. In this paper, we address these problems in the framework of the Bayesian paradigm. To this end, we propose a novel federated Markov Chain Monte Carlo algorithm, referred to as Quantised Langevin Stochastic Dynamics which may be seen as an extension to the FL setting of Stochastic Gradient Langevin Dynamics, which handles the communication bottleneck using gradient compression. To improve performance, we then introduce variance reduction techniques, which lead to two improved versions coined \texttt{QLSD}${}^{\star }$ and \texttt{QLSD}${}^{++}$. We give both non-asymptotic and asymptotic convergence guarantees for the proposed algorithms. We illustrate their performances using various Bayesian Federated Learning benchmarks.

References in corpus (8)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• On Calibration of Modern Neural Networks
• A Field Guide to Federated Optimization
• What Are Bayesian Neural Network Posteriors Really Like?
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
• Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs
• Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification
• DG-LMC: A Turn-key and Scalable Synchronous Distributed MCMC Algorithm via Langevin Monte Carlo within Gibbs