Specialized federated learning using a mixture of experts

Abstract

In federated learning, clients share a global model that has been trained on decentralized local client data. Although federated learning shows significant promise as a key approach when data cannot be shared or centralized, current methods show limited privacy properties and have shortcomings when applied to common real-world scenarios, especially when client data is heterogeneous. In this paper, we propose an alternative method to learn a personalized model for each client in a federated setting, with greater generalization abilities than previous methods. To achieve this personalization we propose a federated learning framework using a mixture of experts to combine the specialist nature of a locally trained model with the generalist knowledge of a global model. We evaluate our method on a variety of datasets with different levels of data heterogeneity, and our results show that the mixture of experts model is better suited as a personalized model for devices in these settings, outperforming both fine-tuned global models and local specialists.

arXiv comment8 pages, 6 figures

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Federated Learning for Mobile Keyboard Prediction
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Federated Learning with Personalization Layers
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Domain Adaptation: Learning Bounds and Algorithms
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Personalized Federated Learning: A Meta-Learning Approach
• Adaptive Personalized Federated Learning
• Federated Learning of a Mixture of Global and Local Models
• Bayesian Nonparametric Federated Learning of Neural Networks
• Private Federated Learning with Domain Adaptation

Cited by in corpus (4)

• Decentralized federated learning of deep neural networks on non-iid data
• Practical and Secure Federated Recommendation with Personalized Masks
• Demystifying the Effects of Non-Independence in Federated Learning
• FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learning