FedSAE: A Novel Self-Adaptive Federated Learning Framework in Heterogeneous Systems

Abstract

Federated Learning (FL) is a novel distributed machine learning which allows thousands of edge devices to train model locally without uploading data concentrically to the server. But since real federated settings are resource-constrained, FL is encountered with systems heterogeneity which causes a lot of stragglers directly and then leads to significantly accuracy reduction indirectly. To solve the problems caused by systems heterogeneity, we introduce a novel self-adaptive federated framework FedSAE which adjusts the training task of devices automatically and selects participants actively to alleviate the performance degradation. In this work, we 1) propose FedSAE which leverages the complete information of devices' historical training tasks to predict the affordable training workloads for each device. In this way, FedSAE can estimate the reliability of each device and self-adaptively adjust the amount of training load per client in each round. 2) combine our framework with Active Learning to self-adaptively select participants. Then the framework accelerates the convergence of the global model. In our framework, the server evaluates devices' value of training based on their training loss. Then the server selects those clients with bigger value for the global model to reduce communication overhead. The experimental result indicates that in a highly heterogeneous system, FedSAE converges faster than FedAvg, the vanilla FL framework. Furthermore, FedSAE outperforms than FedAvg on several federated datasets - FedSAE improves test accuracy by 26.7% and reduces stragglers by 90.3% on average.

arXiv commentThis paper will be presented at IJCNN 2021

References in corpus (21)

• TensorFlow: A system for large-scale machine learning
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Challenges, Methods, and Future Directions
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
• Federated Learning for Mobile Keyboard Prediction
• On the Convergence of FedAvg on Non-IID Data
• Towards Federated Learning at Scale: System Design
• Learning Differentially Private Recurrent Language Models
• Federated Optimization in Heterogeneous Networks
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• Fair Resource Allocation in Federated Learning
• LEAF: A Benchmark for Federated Settings
• Agnostic Federated Learning
• Astraea: Self-balancing Federated Learning for Improving Classification Accuracy of Mobile Deep Learning Applications
• Communication Compression for Decentralized Training
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Active Federated Learning
• Anytime MiniBatch: Exploiting Stragglers in Online Distributed Optimization

Cited by in corpus (3)

• Towards Personalized Federated Learning
• FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measure
• Resource-Efficient Federated Learning