Oort: Efficient Federated Learning via Guided Participant Selection

Abstract

Federated Learning (FL) is an emerging direction in distributed machine learning (ML) that enables in-situ model training and testing on edge data. Despite having the same end goals as traditional ML, FL executions differ significantly in scale, spanning thousands to millions of participating devices. As a result, data characteristics and device capabilities vary widely across clients. Yet, existing efforts randomly select FL participants, which leads to poor model and system efficiency. In this paper, we propose Oort to improve the performance of federated training and testing with guided participant selection. With an aim to improve time-to-accuracy performance in model training, Oort prioritizes the use of those clients who have both data that offers the greatest utility in improving model accuracy and the capability to run training quickly. To enable FL developers to interpret their results in model testing, Oort enforces their requirements on the distribution of participant data while improving the duration of federated testing by cherry-picking clients. Our evaluation shows that, compared to existing participant selection mechanisms, Oort improves time-to-accuracy performance by 1.2x-14.1x and final model accuracy by 1.3%-9.8%, while efficiently enforcing developer-specified model testing criteria at the scale of millions of clients.

References in corpus (21)

• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Federated Learning for Mobile Keyboard Prediction
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Differentially Private Federated Learning: A Client Level Perspective
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Three Approaches for Personalization with Applications to Federated Learning
• Agnostic Federated Learning
• Don't Use Large Mini-Batches, Use Local SGD
• Federated Adversarial Domain Adaptation
• Prio: Private, Robust, and Scalable Computation of Aggregate Statistics
• Distributed Mean Estimation with Limited Communication
• Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies
• Is There a Trade-Off Between Fairness and Accuracy? A Perspective Using Mismatched Hypothesis Testing
• Biased Importance Sampling for Deep Neural Network Training
• Federated Learning with Only Positive Labels
• Federated Learning for Ranking Browser History Suggestions
• Federated Learning of N-gram Language Models
• Training Production Language Models without Memorizing User Data

Cited by in corpus (6)

• Federated Few-Shot Learning for Mobile NLP
• Ed-Fed: A generic federated learning framework with resource-aware client selection for edge devices
• Accelerating MoE Model Inference with Expert Sharding
• Practical Federated Learning without a Server
• Canoe : A System for Collaborative Learning for Neural Nets
• FedOptima: Optimizing Resource Utilization in Federated Learning