Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data

Abstract

Federated learning enables multiple clients to collaboratively learn a global model by periodically aggregating the clients' models without transferring the local data. However, due to the heterogeneity of the system and data, many approaches suffer from the "client-drift" issue that could significantly slow down the convergence of the global model training. As clients perform local updates on heterogeneous data through heterogeneous systems, their local models drift apart. To tackle this issue, one intuitive idea is to guide the local model training by the global teachers, i.e., past global models, where each client learns the global knowledge from past global models via adaptive knowledge distillation techniques. Coming from these insights, we propose a novel approach for heterogeneous federated learning, namely FedGKD, which fuses the knowledge from historical global models for local training to alleviate the "client-drift" issue. In this paper, we evaluate FedGKD with extensive experiments on various CV/NLP datasets (i.e., CIFAR-10/100, Tiny-ImageNet, AG News, SST5) and different heterogeneous settings. The proposed method is guaranteed to converge under common assumptions, and achieves superior empirical accuracy in fewer communication runs than five state-of-the-art methods.

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• On the Convergence of FedAvg on Non-IID Data
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Federated Optimization in Heterogeneous Networks
• A Field Guide to Federated Optimization
• Federated Learning Based on Dynamic Regularization
• Communication-Efficient Federated Distillation

Cited by in corpus (4)

• FedICT: Federated Multi-task Distillation for Multi-access Edge Computing
• UNIDEAL: Curriculum Knowledge Distillation Federated Learning
• Federated Learning on Non-iid Data via Local and Global Distillation
• Federated Modality-specific Encoders and Partially Personalized Fusion Decoder for Multimodal Brain Tumor Segmentation