Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies

Abstract

The increasing demand for privacy-preserving collaborative learning has given rise to a new computing paradigm called federated learning (FL), in which clients collaboratively train a machine learning (ML) model without revealing their private training data. Given an acceptable level of privacy guarantee, the goal of FL is to minimize the time-to-accuracy of model training. Compared with distributed ML in data centers, there are four distinct challenges to achieving short time-to-accuracy in FL training, namely the lack of information for optimization, the tradeoff between statistical and system utility, client heterogeneity, and large configuration space. In this paper, we survey recent works in addressing these challenges and present them following a typical training workflow through three phases: client selection, configuration, and reporting. We also review system works including measurement studies and benchmarking tools that aim to support FL developers.

arXiv commentThis article has been accepted for publication in IEEE Transactions on Big Data. This is the author's version which has not been fully edited and content may change prior to final publication

References in corpus (33)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Federated Learning: Challenges, Methods, and Future Directions
• Federated Learning: Strategies for Improving Communication Efficiency
• Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
• Federated Learning for Mobile Keyboard Prediction
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Secure Federated Transfer Learning
• Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation
• The Hidden Vulnerability of Distributed Learning in Byzantium
• Sparse Communication for Distributed Gradient Descent
• FedMD: Heterogenous Federated Learning via Model Distillation
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• FedML: A Research Library and Benchmark for Federated Machine Learning
• A generic framework for privacy preserving deep learning
• Three Approaches for Personalization with Applications to Federated Learning
• A Survey of Privacy Attacks in Machine Learning
• A Field Guide to Federated Optimization
• Federated Learning for Emoji Prediction in a Mobile Keyboard
• FastSecAgg: Scalable Secure Aggregation for Privacy-Preserving Federated Learning
• IBM Federated Learning: an Enterprise Framework White Paper V0.1
• Abnormal Client Behavior Detection in Federated Learning
• Federated Learning Of Out-Of-Vocabulary Words
• Accelerating Federated Learning over Reliability-Agnostic Clients in Mobile Edge Computing Systems
• Coded Computing for Low-Latency Federated Learning over Wireless Edge Networks
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Multi-Participant Multi-Class Vertical Federated Learning
• A Quasi-Newton Method Based Vertical Federated Learning Framework for Logistic Regression
• The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems
• Understanding and Benchmarking the Impact of GDPR on Database Systems
• FLASHE: Additively Symmetric Homomorphic Encryption for Cross-Silo Federated Learning
• Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search
• Privacy is What We Care About: Experimental Investigation of Federated Learning on Edge Devices
• Sparsified Secure Aggregation for Privacy-Preserving Federated Learning

Cited by in corpus (3)

• Federated Learning Survey: A Multi-Level Taxonomy of Aggregation Techniques, Experimental Insights, and Future Frontiers
• A Safe Deep Reinforcement Learning Approach for Energy Efficient Federated Learning in Wireless Communication Networks
• Dordis: Efficient Federated Learning with Dropout-Resilient Differential Privacy