FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measure

Abstract

Federated Learning (FL) enables the multiple participating devices to collaboratively contribute to a global neural network model while keeping the training data locally. Unlike the centralized training setting, the non-IID and imbalanced (statistical heterogeneity) training data of FL is distributed in the federated network, which will increase the divergences between the local models and global model, further degrading performance. In this paper, we propose a novel clustered federated learning (CFL) framework FedGroup, in which we 1) group the training of clients based on the similarities between the clients' optimization directions for high training performance; 2) construct a new data-driven distance measure to improve the efficiency of the client clustering procedure. 3) implement a newcomer device cold start mechanism based on the auxiliary global model for framework scalability and practicality. FedGroup can achieve improvements by dividing joint optimization into groups of sub-optimization and can be combined with FL optimizer FedProx. The convergence and complexity are analyzed to demonstrate the efficiency of our proposed framework. We also evaluate FedGroup and FedGrouProx (combined with FedProx) on several open datasets and made comparisons with related CFL frameworks. The results show that FedGroup can significantly improve absolute test accuracy by +14.1% on FEMNIST compared to FedAvg. +3.4% on Sentiment140 compared to FedProx, +6.9% on MNIST compared to FeSEM.

arXiv commentThis work will be presented at IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) 2021. NOTE: This revision contains a crucial correction of the client cold start mechanism, please discard all previous manuscripts

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: A system for large-scale machine learning
• Federated Learning: Challenges, Methods, and Future Directions
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• On the Convergence of FedAvg on Non-IID Data
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• Agnostic Federated Learning
• Astraea: Self-balancing Federated Learning for Improving Classification Accuracy of Mobile Deep Learning Applications
• FedSAE: A Novel Self-Adaptive Federated Learning Framework in Heterogeneous Systems
• On perfect clustering of high dimension, low sample size data
• CSAFL: A Clustered Semi-Asynchronous Federated Learning Framework
• TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture

Cited by in corpus (8)

• Towards Personalized Federated Learning
• Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges
• Contrastive encoder pre-training-based clustered federated learning for heterogeneous data
• Federated Learning for Commercial Image Sources
• Fed-EC: Bandwidth-Efficient Clustering-Based Federated Learning For Autonomous Visual Robot Navigation
• Federated Learning with Intermediate Representation Regularization
• FedPref: Federated Learning Across Heterogeneous Multi-objective Preferences
• One-Shot Clustering for Federated Learning