Communication-Efficient Federated Distillation

Abstract

Communication constraints are one of the major challenges preventing the wide-spread adoption of Federated Learning systems. Recently, Federated Distillation (FD), a new algorithmic paradigm for Federated Learning with fundamentally different communication properties, emerged. FD methods leverage ensemble distillation techniques and exchange model outputs, presented as soft labels on an unlabeled public data set, between the central server and the participating clients. While for conventional Federated Learning algorithms, like Federated Averaging (FA), communication scales with the size of the jointly trained model, in FD communication scales with the distillation data set size, resulting in advantageous communication properties, especially when large models are trained. In this work, we investigate FD from the perspective of communication efficiency by analyzing the effects of active distillation-data curation, soft-label quantization and delta-coding techniques. Based on the insights gathered from this analysis, we present Compressed Federated Distillation (CFD), an efficient Federated Distillation method. Extensive experiments on Federated image classification and language modeling problems demonstrate that our method can reduce the amount of communication necessary to achieve fixed performance targets by more than two orders of magnitude, when compared to FD and by more than four orders of magnitude when compared with FA.

References in corpus (16)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Language Models are Few-Shot Learners
• Federated Learning: Strategies for Improving Communication Efficiency
• Ternary Weight Networks
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• FedMD: Heterogenous Federated Learning via Model Distillation
• Pointer Sentinel Mixture Models
• EMNIST: an extension of MNIST to handwritten letters
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data
• One-Shot Federated Learning
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• A closer look at the approximation capabilities of neural networks
• Trends and Advancements in Deep Neural Network Communication

Cited by in corpus (5)

• Federated Learning Meets Natural Language Processing: A Survey
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data
• Towards Model Agnostic Federated Learning Using Knowledge Distillation
• Reward-Based 1-bit Compressed Federated Distillation on Blockchain