Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer
arXiv:2109.08119
Abstract
Personalized federated learning (FL) aims to train model(s) that can perform well for individual clients that are highly data and system heterogeneous. Most work in personalized FL, however, assumes using the same model architecture at all clients and increases the communication cost by sending/receiving models. This may not be feasible for realistic scenarios of FL. In practice, clients have highly heterogeneous system-capabilities and limited communication resources. In our work, we propose a personalized FL framework, PerFed-CKT, where clients can use heterogeneous model architectures and do not directly communicate their model parameters. PerFed-CKT uses clustered co-distillation, where clients use logits to transfer their knowledge to other clients that have similar data-distributions. We theoretically show the convergence and generalization properties of PerFed-CKT and empirically show that PerFed-CKT achieves high test accuracy with several orders of magnitude lower communication cost compared to the state-of-the-art personalized FL schemes.
References in corpus (14)
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Towards Federated Learning at Scale: System Design
- Three Approaches for Personalization with Applications to Federated Learning
- On the Convergence of Local Descent Methods in Federated Learning
- A Field Guide to Federated Optimization
- Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
- Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies
- Distilled One-Shot Federated Learning
- Faster On-Device Training Using New Federated Momentum Algorithm
- Online Learning, Stability, and Stochastic Gradient Descent
- Edge Bias in Federated Learning and its Solution by Buffered Knowledge Distillation
- Federated Learning with Positive and Unlabeled Data
- A Closer Look at Codistillation for Distributed Training