QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning

Abstract

The objective of Federated Learning (FL) is to perform statistical inference for data which are decentralised and stored locally on networked clients. FL raises many constraints which include privacy and data ownership, communication overhead, statistical heterogeneity, and partial client participation. In this paper, we address these problems in the framework of the Bayesian paradigm. To this end, we propose a novel federated Markov Chain Monte Carlo algorithm, referred to as Quantised Langevin Stochastic Dynamics which may be seen as an extension to the FL setting of Stochastic Gradient Langevin Dynamics, which handles the communication bottleneck using gradient compression. To improve performance, we then introduce variance reduction techniques, which lead to two improved versions coined \texttt{QLSD}${}^{\star }$ and \texttt{QLSD}${}^{++}$. We give both non-asymptotic and asymptotic convergence guarantees for the proposed algorithms. We illustrate their performances using various Bayesian Federated Learning benchmarks.

References in corpus (23)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• On Calibration of Modern Neural Networks
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• LEAF: A Benchmark for Federated Settings
• Sparsified SGD with Memory
• User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient
• A Field Guide to Federated Optimization
• Variational Federated Multi-Task Learning
• Theory and Applications of Proper Scoring Rules
• What Are Bayesian Neural Network Posteriors Really Like?
• Parallelizing MCMC via Weierstrass Sampler
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
• Parallelizing MCMC with Random Partition Trees
• The promises and pitfalls of Stochastic Gradient Langevin Dynamics
• Uncertainty Quantification and Deep Ensembles
• Partitioned Variational Inference: A unified framework encompassing federated and continual learning
• Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs
• Efficient MCMC Sampling with Dimension-Free Convergence Rate using ADMM-type Splitting
• Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees
• DG-LMC: A Turn-key and Scalable Synchronous Distributed MCMC Algorithm via Langevin Monte Carlo within Gibbs
• Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification