Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matching

Abstract

In federated learning (FL), clients may have diverse objectives, and merging all clients' knowledge into one global model will cause negative transfer to local performance. Thus, clustered FL is proposed to group similar clients into clusters and maintain several global models. In the literature, centralized clustered FL algorithms require the assumption of the number of clusters and hence are not effective enough to explore the latent relationships among clients. In this paper, without assuming the number of clusters, we propose a peer-to-peer (P2P) FL algorithm named PANM. In PANM, clients communicate with peers to adaptively form an effective clustered topology. Specifically, we present two novel metrics for measuring client similarity and a two-stage neighbor matching algorithm based Monte Carlo method and Expectation Maximization under the Gaussian Mixture Model assumption. We have conducted theoretical analyses of PANM on the probability of neighbor estimation and the error gap to the clustered optimum. We have also implemented extensive experiments under both synthetic and real-world clustered heterogeneity. Theoretical analysis and empirical experiments show that the proposed algorithm is superior to the P2P FL counterparts, and it achieves better performance than the centralized cluster FL method. PANM is effective even under extremely low communication budgets.

Published in IEEE Transactions on Big Data, 2022

References in corpus (8)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Deep Domain Confusion: Maximizing for Domain Invariance
• Federated Learning for Internet of Things: A Comprehensive Survey
• Peer-to-peer Federated Learning on Graphs
• Decentralized Federated Learning via Mutual Knowledge Transfer
• Flexible Clustered Federated Learning for Client-Level Data Distribution Shift
• Decentralized federated learning of deep neural networks on non-iid data
• Ensemble Federated Adversarial Training with Non-IID data

Cited by in corpus (2)

• Edge-cloud Collaborative Learning with Federated and Centralized Features
• Federated Source-free Domain Adaptation for Classification: Weighted Cluster Aggregation for Unlabeled Data