FedMD: Heterogenous Federated Learning via Model Distillation

Abstract

Federated learning enables the creation of a powerful centralized model without compromising data privacy of multiple participants. While successful, it does not incorporate the case where each participant independently designs its own model. Due to intellectual property concerns and heterogeneous nature of tasks and data, this is a widespread requirement in applications of federated learning to areas such as health care and AI as a service. In this work, we use transfer learning and knowledge distillation to develop a universal framework that enables federated learning when each agent owns not only their private data, but also uniquely designed models. We test our framework on the MNIST/FEMNIST dataset and the CIFAR10/CIFAR100 dataset and observe fast improvement across all participating models. With 10 distinct participants, the final test accuracy of each model on average receives a 20% gain on top of what's possible without collaboration and is only a few percent lower than the performance each model would have obtained if all private datasets were pooled and made directly available for all participants.

arXiv comment4 pages, 2 figures, NeurIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• Federated Optimization in Heterogeneous Networks
• LEAF: A Benchmark for Federated Settings
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Variational Federated Multi-Task Learning
• Incentive Design for Efficient Federated Learning in Mobile Networks: A Contract Theory Approach
• Semi-Cyclic Stochastic Gradient Descent

Cited by in corpus (62)

• FedKD: Communication Efficient Federated Learning via Knowledge Distillation
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Adaptive Personalized Federated Learning
• Personalized Federated Learning with Moreau Envelopes
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
• Learning to Detect Malicious Clients for Robust Federated Learning
• A Field Guide to Federated Optimization
• A Federated Learning Aggregation Algorithm for Pervasive Computing: Evaluation and Comparison
• No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
• Artificial Intelligence of Things: A Survey
• Distilled One-Shot Federated Learning
• Federated Mutual Learning
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• FedICT: Federated Multi-task Distillation for Multi-access Edge Computing
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• Auto-FedAvg: Learnable Federated Averaging for Multi-Institutional Medical Image Segmentation
• Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies
• FedHe: Heterogeneous Models and Communication-Efficient Federated Learning
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• Personalized Federated Learning for Intelligent IoT Applications: A Cloud-Edge based Framework
• TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture
• Parameterized Knowledge Transfer for Personalized Federated Learning
• Evaluation and comparison of federated learning algorithms for Human Activity Recognition on smartphones
• FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion
• Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space
• Federated Model Distillation with Noise-Free Differential Privacy
• Personalized Federated Learning through Local Memorization
• FedH2L: Federated Learning with Model and Statistical Heterogeneity
• Speeding up Heterogeneous Federated Learning with Sequentially Trained Superclients
• CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning
• A Theoretical Perspective on Differentially Private Federated Multi-task Learning
• FedBoosting: Federated Learning with Gradient Protected Boosting for Text Recognition
• Communication-Efficient Federated Distillation
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• Personalized Federated Learning with Multi-branch Architecture
• Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes
• Federated Learning on Non-IID Data: A Survey
• UNIDEAL: Curriculum Knowledge Distillation Federated Learning
• Federated Learning on Non-iid Data via Local and Global Distillation
• Practical One-Shot Federated Learning for Cross-Silo Setting
• Communication Efficiency Optimization of Federated Learning for Computing and Network Convergence of 6G Networks
• Resource-Constrained Federated Learning with Heterogeneous Labels and Models
• QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
• Multi-Level Analyzation of Imbalance to Resolve Non-IID-Ness in Federated Learning
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• Edge Bias in Federated Learning and its Solution by Buffered Knowledge Distillation
• WAFFLE: Watermarking in Federated Learning
• Federated Loss Exploration for Improved Convergence on Non-IID Data
• Federated Learning with Heterogeneous Labels and Models for Mobile Activity Monitoring
• Source Inference Attacks in Federated Learning
• Federated Learning for Open Banking
• On Large-Cohort Training for Federated Learning
• Inclusive Data Representation in Federated Learning: A Novel Approach Integrating Textual and Visual Prompt
• Collate: Collaborative Neural Network Learning for Latency-Critical Edge Systems
• Privacy-Preserving Self-Taught Federated Learning for Heterogeneous Data
• Vicinal Feature Statistics Augmentation for Federated 3D Medical Volume Segmentation
• ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning
• Exploiting Features and Logits in Heterogeneous Federated Learning
• Towards Explainable Multi-Party Learning: A Contrastive Knowledge Sharing Framework
• A Personalized Federated Learning Algorithm: an Application in Anomaly Detection