Towards Heterogeneous Clients with Elastic Federated Learning

Abstract

Federated learning involves training machine learning models over devices or data silos, such as edge processors or data warehouses, while keeping the data local. Training in heterogeneous and potentially massive networks introduces bias into the system, which is originated from the non-IID data and the low participation rate in reality. In this paper, we propose Elastic Federated Learning (EFL), an unbiased algorithm to tackle the heterogeneity in the system, which makes the most informative parameters less volatile during training, and utilizes the incomplete local updates. It is an efficient and effective algorithm that compresses both upstream and downstream communications. Theoretically, the algorithm has convergence guarantee when training on the non-IID data at the low participation rate. Empirical experiments corroborate the competitive performance of EFL framework on the robustness and the efficiency.

arXiv commentUnder Review

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• On the Convergence of FedAvg on Non-IID Data
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Federated Optimization in Heterogeneous Networks
• Adaptive Personalized Federated Learning
• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• Continual Learning with Deep Generative Replay
• On the Convergence of Local Descent Methods in Federated Learning
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Variance Reduced Local SGD with Lower Communication Complexity
• Neurogenesis Deep Learning
• Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization
• Variance-based Gradient Compression for Efficient Distributed Deep Learning
• Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning