Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions

Abstract

Federated Learning (FL) is a distributed and privacy-preserving machine learning paradigm that coordinates multiple clients to train a model while keeping the raw data localized. However, this traditional FL poses some challenges, including privacy risks, data heterogeneity, communication bottlenecks, and system heterogeneity issues. To tackle these challenges, knowledge distillation (KD) has been widely applied in FL since 2020. KD is a validated and efficacious model compression and enhancement algorithm. The core concept of KD involves facilitating knowledge transfer between models by exchanging logits at intermediate or output layers. These properties make KD an excellent solution for the long-lasting challenges in FL. Up to now, there have been few reviews that summarize and analyze the current trend and methods for how KD can be applied in FL efficiently. This article aims to provide a comprehensive survey of KD-based FL, focusing on addressing the above challenges. First, we provide an overview of KD-based FL, including its motivation, basics, taxonomy, and a comparison with traditional FL and where KD should execute. We also analyze the critical factors in KD-based FL in the appendix, including teachers, knowledge, data, and methods. We discuss how KD can address the challenges in FL, including privacy protection, data heterogeneity, communication efficiency, and personalization. Finally, we discuss the challenges facing KD-based FL algorithms and future research directions. We hope this survey can provide insights and guidance for researchers and practitioners in the FL area.

References in corpus (16)

• Overcoming catastrophic forgetting in neural networks
• Federated Learning: Challenges, Methods, and Future Directions
• Knowledge Distillation: A Survey
• Towards Personalized Federated Learning
• Statistical Aspects of Wasserstein Distances
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation
• Astraea: Self-balancing Federated Learning for Improving Classification Accuracy of Mobile Deep Learning Applications
• Decentralized Federated Learning via Mutual Knowledge Transfer
• No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
• Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space
• AdaBest: Minimizing Client Drift in Federated Learning via Adaptive Bias Estimation
• Vertical Federated Knowledge Transfer via Representation Distillation for Healthcare Collaboration Networks
• Cross-domain Federated Object Detection
• Federated Learning on Non-iid Data via Local and Global Distillation
• Adaptive Self-Distillation for Minimizing Client Drift in Heterogeneous Federated Learning

Cited by in corpus (1)

• From Privacy to Trust in the Agentic Era: A Taxonomy of Challenges in Trustworthy Federated Learning Through the Lens of Trust Report 2.0