Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data

Abstract

This study develops a federated learning (FL) framework overcoming largely incremental communication costs due to model sizes in typical frameworks without compromising model performance. To this end, based on the idea of leveraging an unlabeled open dataset, we propose a distillation-based semi-supervised FL (DS-FL) algorithm that exchanges the outputs of local models among mobile devices, instead of model parameter exchange employed by the typical frameworks. In DS-FL, the communication cost depends only on the output dimensions of the models and does not scale up according to the model size. The exchanged model outputs are used to label each sample of the open dataset, which creates an additionally labeled dataset. Based on the new dataset, local models are further trained, and model performance is enhanced owing to the data augmentation effect. We further highlight that in DS-FL, the heterogeneity of the devices' dataset leads to ambiguous of each data sample and lowing of the training convergence. To prevent this, we propose entropy reduction averaging, where the aggregated model outputs are intentionally sharpened. Moreover, extensive experiments show that DS-FL reduces communication costs up to 99% relative to those of the FL benchmark while achieving similar or higher classification accuracy.

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Revisiting Distributed Synchronous SGD
• Federated Learning for 6G Communications: Challenges, Methods, and Future Directions
• Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• Coded Federated Learning
• Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective
• Lottery Hypothesis based Unsupervised Pre-training for Model Compression in Federated Learning

Cited by in corpus (14)

• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Data
• FedICT: Federated Multi-task Distillation for Multi-access Edge Computing
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion
• Parameterized Knowledge Transfer for Personalized Federated Learning
• Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space
• GFL: A Decentralized Federated Learning Framework Based On Blockchain
• Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models
• Communication-Efficient Federated Distillation
• SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
• Efficient Ring-topology Decentralized Federated Learning with Deep Generative Models for Industrial Artificial Intelligent
• Federated Learning with Positive and Unlabeled Data
• Reward-Based 1-bit Compressed Federated Distillation on Blockchain