Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms

Abstract

Federated learning is typically approached as an optimization problem, where the goal is to minimize a global loss function by distributing computation across client devices that possess local data and specify different parts of the global objective. We present an alternative perspective and formulate federated learning as a posterior inference problem, where the goal is to infer a global posterior distribution by having client devices each infer the posterior of their local data. While exact inference is often intractable, this perspective provides a principled way to search for global optima in federated settings. Further, starting with the analysis of federated quadratic objectives, we develop a computation- and communication-efficient approximate posterior inference algorithm -- federated posterior averaging (FedPA). Our algorithm uses MCMC for approximate inference of local posteriors on the clients and efficiently communicates their statistics to the server, where the latter uses them to refine a global estimate of the posterior mode. Finally, we show that FedPA generalizes federated averaging (FedAvg), can similarly benefit from adaptive optimizers, and yields state-of-the-art results on four realistic and challenging benchmarks, converging faster, to better optima.

arXiv commentICLR 2021. Code: https://github.com/alshedivat/fedpa

References in corpus (18)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• Federated Learning for Mobile Keyboard Prediction
• On the Convergence of FedAvg on Non-IID Data
• Towards Federated Learning at Scale: System Design
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Shrinkage Algorithms for MMSE Covariance Estimation
• Federated Optimization in Heterogeneous Networks
• LEAF: A Benchmark for Federated Settings
• Adaptive Federated Optimization
• Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlo
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• The Non-IID Data Quagmire of Decentralized Machine Learning
• FedSplit: An algorithmic framework for fast federated optimization
• Minibatch vs Local SGD for Heterogeneous Distributed Learning
• Is Local SGD Better than Minibatch SGD?
• On the Outsized Importance of Learning Rates in Local Update Methods
• On the Convergence of SGD with Biased Gradients

Cited by in corpus (1)

• An Expectation-Maximization Perspective on Federated Learning