Contrastive encoder pre-training-based clustered federated learning for heterogeneous data

Abstract

Federated learning (FL) is a promising approach that enables distributed clients to collaboratively train a global model while preserving their data privacy. However, FL often suffers from data heterogeneity problems, which can significantly affect its performance. To address this, clustered federated learning (CFL) has been proposed to construct personalized models for different client clusters. One effective client clustering strategy is to allow clients to choose their own local models from a model pool based on their performance. However, without pre-trained model parameters, such a strategy is prone to clustering failure, in which all clients choose the same model. Unfortunately, collecting a large amount of labeled data for pre-training can be costly and impractical in distributed environments. To overcome this challenge, we leverage self-supervised contrastive learning to exploit unlabeled data for the pre-training of FL systems. Together, self-supervised pre-training and client clustering can be crucial components for tackling the data heterogeneity issues of FL. Leveraging these two crucial strategies, we propose contrastive pre-training-based clustered federated learning (CP-CFL) to improve the model convergence and overall performance of FL systems. In this work, we demonstrate the effectiveness of CP-CFL through extensive experiments in heterogeneous FL settings, and present various interesting observations.

arXiv commentPublished in Neural Networks

References in corpus (30)

• Auto-Encoding Variational Bayes
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Federated Learning: Challenges, Methods, and Future Directions
• Bootstrap your own latent: A new approach to self-supervised Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Unsupervised Representation Learning by Predicting Image Rotations
• Federated Learning for Mobile Keyboard Prediction
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Learning Representations by Maximizing Mutual Information Across Views
• Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocation
• Three Approaches for Personalization with Applications to Federated Learning
• Multi-Center Federated Learning: Clients Clustering for Better Personalization
• Multi-Center Federated Learning: Clients Clustering for Better Personalization
• CLEAR: Contrastive Learning for Sentence Representation
• Federated Learning for Emoji Prediction in a Mobile Keyboard
• TCLR: Temporal Contrastive Learning for Video Representation
• Robust Federated Learning in a Heterogeneous Environment
• FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measure
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Federated Unsupervised Representation Learning
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Divergence-aware Federated Self-Supervised Learning
• Attention on Personalized Clinical Decision Support System: Federated Learning Approach
• FedCon: A Contrastive Framework for Federated Semi-Supervised Learning
• Federated Self-supervised Learning for Heterogeneous Clients
• Federated Learning with Intermediate Representation Regularization
• Toward Multiple Federated Learning Services Resource Sharing in Mobile Edge Networks

Cited by in corpus (1)

• Federated Learning with Intermediate Representation Regularization