FedCM: Federated Learning with Client-level Momentum
arXiv:2106.10874
Abstract
Federated Learning is a distributed machine learning approach which enables model training without data sharing. In this paper, we propose a new federated learning algorithm, Federated Averaging with Client-level Momentum (FedCM), to tackle problems of partial participation and client heterogeneity in real-world federated learning applications. FedCM aggregates global gradient information in previous communication rounds and modifies client gradient descent with a momentum-like term, which can effectively correct the bias and improve the stability of local SGD. We provide theoretical analysis to highlight the benefits of FedCM. We also perform extensive empirical studies and demonstrate that FedCM achieves superior performance in various tasks and is robust to different levels of client numbers, participation rate and client heterogeneity.
References in corpus (24)
- Adam: A Method for Stochastic Optimization
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning: Strategies for Improving Communication Efficiency
- Federated Learning with Non-IID Data
- On the Convergence of FedAvg on Non-IID Data
- SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
- Federated Optimization in Heterogeneous Networks
- Communication Efficient Distributed Optimization using an Approximate Newton-type Method
- Improving Federated Learning Personalization via Model Agnostic Meta Learning
- Personalized Federated Learning with Moreau Envelopes
- Local SGD Converges Fast and Communicates Little
- Adaptive Federated Optimization
- Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
- Variance Reduced Local SGD with Lower Communication Complexity
- FedSplit: An algorithmic framework for fast federated optimization
- SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
- Lower Bounds and Optimal Algorithms for Personalized Federated Learning
- Faster On-Device Training Using New Federated Momentum Algorithm
- Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
- Federated Accelerated Stochastic Gradient Descent
- Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
- Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data
- FedDANE: A Federated Newton-Type Method