An Operator Splitting View of Federated Learning

Abstract

Over the past few years, the federated learning ($\text{FL}$) community has witnessed a proliferation of new $\text{FL}$ algorithms. However, our understating of the theory of $\text{FL}$ is still fragmented, and a thorough, formal comparison of these algorithms remains elusive. Motivated by this gap, we show that many of the existing $\text{FL}$ algorithms can be understood from an operator splitting point of view. This unification allows us to compare different algorithms with ease, to refine previous convergence results and to uncover new algorithmic variants. In particular, our analysis reveals the vital role played by the step size in $\text{FL}$ algorithms. The unification also leads to a streamlined and economic way to accelerate $\text{FL}$ algorithms, without incurring any communication overhead. We perform numerical experiments on both convex and nonconvex models to validate our findings.

30 pages, 28 figures

References in corpus (20)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• How To Backdoor Federated Learning
• Can You Really Backdoor Federated Learning?
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• LEAF: A Benchmark for Federated Settings
• Agnostic Federated Learning
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
• Bayesian Nonparametric Federated Learning of Neural Networks
• Adaptive Federated Optimization
• Federated Learning with Matched Averaging
• FedSplit: An algorithmic framework for fast federated optimization
• Anderson Accelerated Douglas-Rachford Splitting
• First Analysis of Local GD on Heterogeneous Data
• Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization
• Faster On-Device Training Using New Federated Momentum Algorithm
• Federated Learning Meets Multi-objective Optimization
• From Local SGD to Local Fixed-Point Methods for Federated Learning

Cited by in corpus (1)

• Iterated Vector Fields and Conservatism, with Applications to Federated Learning