FedCM: Federated Learning with Client-level Momentum
arXiv:2106.10874
Abstract
Federated Learning is a distributed machine learning approach which enables model training without data sharing. In this paper, we propose a new federated learning algorithm, Federated Averaging with Client-level Momentum (FedCM), to tackle problems of partial participation and client heterogeneity in real-world federated learning applications. FedCM aggregates global gradient information in previous communication rounds and modifies client gradient descent with a momentum-like term, which can effectively correct the bias and improve the stability of local SGD. We provide theoretical analysis to highlight the benefits of FedCM. We also perform extensive empirical studies and demonstrate that FedCM achieves superior performance in various tasks and is robust to different levels of client numbers, participation rate and client heterogeneity.
References in corpus (6)
- Variance Reduced Local SGD with Lower Communication Complexity
- FedSplit: An algorithmic framework for fast federated optimization
- SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
- Lower Bounds and Optimal Algorithms for Personalized Federated Learning
- Faster On-Device Training Using New Federated Momentum Algorithm
- FedDANE: A Federated Newton-Type Method