From Local SGD to Local Fixed-Point Methods for Federated Learning

Abstract

Most algorithms for solving optimization problems or finding saddle points of convex-concave functions are fixed-point algorithms. In this work we consider the generic problem of finding a fixed point of an average of operators, or an approximation thereof, in a distributed setting. Our work is motivated by the needs of federated learning. In this context, each local operator models the computations done locally on a mobile device. We investigate two strategies to achieve such a consensus: one based on a fixed number of local steps, and the other based on randomized computations. In both cases, the goal is to limit communication of the locally-computed variables, which is often the bottleneck in distributed frameworks. We perform convergence analysis of both methods and conduct a number of experiments highlighting the benefits of our approach.

arXiv commentAccepted to ICML 2020

References in corpus (6)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• On the Convergence of Local Descent Methods in Federated Learning
• First Analysis of Local GD on Heterogeneous Data
• Gradient Descent with Compressed Iterates
• Distributed Fixed Point Methods with Compressed Iterates

Cited by in corpus (5)

• FedML: A Research Library and Benchmark for Federated Machine Learning
• Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies
• Local Adaptivity in Federated Learning: Convergence and Consistency
• FedEval: A Holistic Evaluation Framework for Federated Learning
• An Operator Splitting View of Federated Learning