Auxo: Efficient Federated Learning via Scalable Client Clustering

Abstract

Federated learning (FL) is an emerging machine learning (ML) paradigm that enables heterogeneous edge devices to collaboratively train ML models without revealing their raw data to a logically centralized server. However, beyond the heterogeneous device capacity, FL participants often exhibit differences in their data distributions, which are not independent and identically distributed (Non-IID). Many existing works present point solutions to address issues like slow convergence, low final accuracy, and bias in FL, all stemming from client heterogeneity. In this paper, we explore an additional layer of complexity to mitigate such heterogeneity by grouping clients with statistically similar data distributions (cohorts). We propose Auxo to gradually identify such cohorts in large-scale, low-availability, and resource-constrained FL populations. Auxo then adaptively determines how to train cohort-specific models in order to achieve better model performance and ensure resource efficiency. Our extensive evaluations show that, by identifying cohorts with smaller heterogeneity and performing efficient cohort-based training, Auxo boosts various existing FL solutions in terms of final accuracy (2.1% - 8.2%), convergence time (up to 2.2x), and model bias (4.8% - 53.8%).

arXiv comment18 pages

References in corpus (23)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Federated Learning with Non-IID Data
• RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• On the Convergence of FedAvg on Non-IID Data
• How To Backdoor Federated Learning
• Learning Differentially Private Recurrent Language Models
• Poisoning Attacks against Support Vector Machines
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Active Learning for Convolutional Neural Networks: A Core-Set Approach
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Fair Resource Allocation in Federated Learning
• Multi-Center Federated Learning: Clients Clustering for Better Personalization
• Multi-Center Federated Learning: Clients Clustering for Better Personalization
• Ditto: Fair and Robust Federated Learning Through Personalization
• Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent
• Flexible Clustered Federated Learning for Client-Level Data Distribution Shift
• FLaaS: Federated Learning as a Service
• Robust Federated Learning: The Case of Affine Distribution Shifts
• FedScale: Benchmarking Model and System Performance of Federated Learning at Scale
• Motley: Benchmarking Heterogeneity and Personalization in Federated Learning
• Pisces: Efficient Federated Learning via Guided Asynchronous Training
• FLINT: A Platform for Federated Learning Integration