FedMD: Heterogenous Federated Learning via Model Distillation

Abstract

Federated learning enables the creation of a powerful centralized model without compromising data privacy of multiple participants. While successful, it does not incorporate the case where each participant independently designs its own model. Due to intellectual property concerns and heterogeneous nature of tasks and data, this is a widespread requirement in applications of federated learning to areas such as health care and AI as a service. In this work, we use transfer learning and knowledge distillation to develop a universal framework that enables federated learning when each agent owns not only their private data, but also uniquely designed models. We test our framework on the MNIST/FEMNIST dataset and the CIFAR10/CIFAR100 dataset and observe fast improvement across all participating models. With 10 distinct participants, the final test accuracy of each model on average receives a 20% gain on top of what's possible without collaboration and is only a few percent lower than the performance each model would have obtained if all private datasets were pooled and made directly available for all participants.

4 pages, 2 figures, NeurIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality

References in corpus (2)

• Distilling the Knowledge in a Neural Network
• Semi-Cyclic Stochastic Gradient Descent

Cited by in corpus (20)

• Learning to Detect Malicious Clients for Robust Federated Learning
• A Field Guide to Federated Optimization
• A Federated Learning Aggregation Algorithm for Pervasive Computing: Evaluation and Comparison
• Federated Mutual Learning
• Auto-FedAvg: Learnable Federated Averaging for Multi-Institutional Medical Image Segmentation
• Evaluation and comparison of federated learning algorithms for Human Activity Recognition on smartphones
• Parameterized Knowledge Transfer for Personalized Federated Learning
• A Theoretical Perspective on Differentially Private Federated Multi-task Learning
• Communication-Efficient Federated Distillation
• Federated Learning on Non-IID Data: A Survey
• Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes
• Resource-Constrained Federated Learning with Heterogeneous Labels and Models
• Source Inference Attacks in Federated Learning
• Federated Learning with Heterogeneous Labels and Models for Mobile Activity Monitoring
• On Large-Cohort Training for Federated Learning
• Federated Learning for Open Banking
• ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning
• Privacy-Preserving Self-Taught Federated Learning for Heterogeneous Data
• Towards Explainable Multi-Party Learning: A Contrastive Knowledge Sharing Framework
• A Personalized Federated Learning Algorithm: an Application in Anomaly Detection