Decentralized Bayesian Learning over Graphs

Abstract

We propose a decentralized learning algorithm over a general social network. The algorithm leaves the training data distributed on the mobile devices while utilizing a peer to peer model aggregation method. The proposed algorithm allows agents with local data to learn a shared model explaining the global training data in a decentralized fashion. The proposed algorithm can be viewed as a Bayesian and peer-to-peer variant of federated learning in which each agent keeps a "posterior probability distribution" over a global model parameters. The agent update its "posterior" based on 1) the local training data and 2) the asynchronous communication and model aggregation with their 1-hop neighbors. This Bayesian formulation allows for a systematic treatment of model aggregation over any arbitrary connected graph. Furthermore, it provides strong analytic guarantees on converge in the realizable case as well as a closed form characterization of the rate of convergence. We also show that our methodology can be combined with efficient Bayesian inference techniques to train Bayesian neural networks in a decentralized manner. By empirical studies we show that our theoretical analysis can guide the design of network/social interactions and data partitioning to achieve convergence.

References in corpus (13)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Variational Dropout and the Local Reparameterization Trick
• Don't Use Large Mini-Batches, Use Local SGD
• D${}^2$: Decentralized Training over Decentralized Data
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Streaming Variational Bayes
• How to scale distributed deep learning?
• Parle: parallelizing stochastic gradient descent

Cited by in corpus (6)

• FedML: A Research Library and Benchmark for Federated Machine Learning
• Federated Learning for Healthcare Informatics
• Decentralized Stochastic Gradient Langevin Dynamics and Hamiltonian Monte Carlo
• A(DP)${}^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy
• Byzantine-resilient Decentralized Stochastic Gradient Descent
• Bristle: Decentralized Federated Learning in Byzantine, Non-i.i.d. Environments