UNIDEAL: Curriculum Knowledge Distillation Federated Learning

Abstract

Federated Learning (FL) has emerged as a promising approach to enable collaborative learning among multiple clients while preserving data privacy. However, cross-domain FL tasks, where clients possess data from different domains or distributions, remain a challenging problem due to the inherent heterogeneity. In this paper, we present UNIDEAL, a novel FL algorithm specifically designed to tackle the challenges of cross-domain scenarios and heterogeneous model architectures. The proposed method introduces Adjustable Teacher-Student Mutual Evaluation Curriculum Learning, which significantly enhances the effectiveness of knowledge distillation in FL settings. We conduct extensive experiments on various datasets, comparing UNIDEAL with state-of-the-art baselines. Our results demonstrate that UNIDEAL achieves superior performance in terms of both model accuracy and communication efficiency. Additionally, we provide a convergence analysis of the algorithm, showing a convergence rate of O(1/T) under non-convex conditions.

Submitted to ICASSP 2024

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Towards Personalized Federated Learning
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation
• FedMD: Heterogenous Federated Learning via Model Distillation
• FedALA: Adaptive Local Aggregation for Personalized Federated Learning
• Federated User Representation Learning
• Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data