Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies

Abstract

The increasing demand for privacy-preserving collaborative learning has given rise to a new computing paradigm called federated learning (FL), in which clients collaboratively train a machine learning (ML) model without revealing their private training data. Given an acceptable level of privacy guarantee, the goal of FL is to minimize the time-to-accuracy of model training. Compared with distributed ML in data centers, there are four distinct challenges to achieving short time-to-accuracy in FL training, namely the lack of information for optimization, the tradeoff between statistical and system utility, client heterogeneity, and large configuration space. In this paper, we survey recent works in addressing these challenges and present them following a typical training workflow through three phases: client selection, configuration, and reporting. We also review system works including measurement studies and benchmarking tools that aim to support FL developers.

This article has been accepted for publication in IEEE Transactions on Big Data. This is the author's version which has not been fully edited and content may change prior to final publication

References in corpus (13)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• FedMD: Heterogenous Federated Learning via Model Distillation
• A Field Guide to Federated Optimization
• Federated Learning for Emoji Prediction in a Mobile Keyboard
• FastSecAgg: Scalable Secure Aggregation for Privacy-Preserving Federated Learning
• Abnormal Client Behavior Detection in Federated Learning
• IBM Federated Learning: an Enterprise Framework White Paper V0.1
• Federated Learning Of Out-Of-Vocabulary Words
• Multi-Participant Multi-Class Vertical Federated Learning
• A Quasi-Newton Method Based Vertical Federated Learning Framework for Logistic Regression
• FLASHE: Additively Symmetric Homomorphic Encryption for Cross-Silo Federated Learning
• Privacy is What We Care About: Experimental Investigation of Federated Learning on Edge Devices
• Sparsified Secure Aggregation for Privacy-Preserving Federated Learning

Cited by in corpus (3)

• Federated Learning Survey: A Multi-Level Taxonomy of Aggregation Techniques, Experimental Insights, and Future Frontiers
• A Safe Deep Reinforcement Learning Approach for Energy Efficient Federated Learning in Wireless Communication Networks
• Dordis: Efficient Federated Learning with Dropout-Resilient Differential Privacy