Distilled One-Shot Federated Learning

Abstract

Current federated learning algorithms take tens of communication rounds transmitting unwieldy model weights under ideal circumstances and hundreds when data is poorly distributed. Inspired by recent work on dataset distillation and distributed one-shot learning, we propose Distilled One-Shot Federated Learning (DOSFL) to significantly reduce the communication cost while achieving comparable performance. In just one round, each client distills their private dataset, sends the synthetic data (e.g. images or sentences) to the server, and collectively trains a global model. The distilled data look like noise and are only useful to the specific model weights, i.e., become useless after the model updates. With this weight-less and gradient-less design, the total communication cost of DOSFL is up to three orders of magnitude less than FedAvg while preserving between 93% to 99% performance of a centralized counterpart. Afterwards, clients could switch to traditional methods such as FedAvg to finetune the last few percent to fit personalized local models with local datasets. Through comprehensive experiments, we show the accuracy and communication performance of DOSFL on both vision and language tasks with different models including CNN, LSTM, Transformer, etc. We demonstrate that an eavesdropping attacker cannot properly train a good model using the leaked distilled data, without knowing the initial model weights. DOSFL serves as an inexpensive method to quickly converge on a performant pre-trained model with less than 0.1% communication cost of traditional methods.

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• On the Convergence of FedAvg on Non-IID Data
• How To Backdoor Federated Learning
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation
• FedMD: Heterogenous Federated Learning via Model Distillation
• Federated Optimization in Heterogeneous Networks
• Asynchronous Federated Optimization
• FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
• Practical Secure Aggregation for Federated Learning on User-Held Data
• Overcoming Forgetting in Federated Learning on Non-IID Data
• One-Shot Federated Learning
• Adaptive Federated Optimization
• Soft-Label Dataset Distillation and Text Dataset Distillation
• One-Shot Federated Learning: Theoretical Limits and Algorithms to Achieve Them
• Accelerating Federated Learning via Momentum Gradient Descent
• Server Averaging for Federated Learning
• Order Optimal One-Shot Distributed Learning

Cited by in corpus (15)

• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Federated Learning: Opportunities and Challenges
• A Comprehensive Survey of Dataset Distillation
• Dataset Condensation with Gradient Matching
• Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources
• Federated Learning Meets Natural Language Processing: A Survey
• One-shot Federated Learning without Server-side Training
• Practical Vertical Federated Learning with Unsupervised Representation Learning
• Parameterized Knowledge Transfer for Personalized Federated Learning
• Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer
• Practical One-Shot Federated Learning for Cross-Silo Setting
• Robust Federated Learning with Attack-Adaptive Aggregation
• Distributed Machine Learning for Wireless Communication Networks: Techniques, Architectures, and Applications
• TOFU: Towards Obfuscated Federated Updates by Encoding Weight Updates into Gradients from Proxy Data