Federated Mixture of Experts
arXiv:2107.06724
Abstract
Federated learning (FL) has emerged as the predominant approach for collaborative training of neural network models across multiple users, without the need to gather the data at a central location. One of the important challenges in this setting is data heterogeneity, i.e. different users have different data characteristics. For this reason, training and using a single global model might be suboptimal when considering the performance of each of the individual user's data. In this work, we tackle this problem via Federated Mixture of Experts, FedMix, a framework that allows us to train an ensemble of specialized models. FedMix adaptively selects and trains a user-specific selection of the ensemble members. We show that users with similar data characteristics select the same members and therefore share statistical strength while mitigating the effect of non-i.i.d data. Empirically, we show through an extensive experimental evaluation that FedMix improves performance compared to using a single global model across a variety of different sources of non-i.i.d.-ness.
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Towards Federated Learning at Scale: System Design
- Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
- On First-Order Meta-Learning Algorithms
- Federated Optimization in Heterogeneous Networks
- Improving Federated Learning Personalization via Model Agnostic Meta Learning
- Personalized Federated Learning: A Meta-Learning Approach
- Fair Resource Allocation in Federated Learning
- Adaptive Personalized Federated Learning
- Three Approaches for Personalization with Applications to Federated Learning
- LEAF: A Benchmark for Federated Settings
- Federated Meta-Learning with Fast Convergence and Efficient Communication
- Agnostic Federated Learning
- Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
- Think Locally, Act Globally: Federated Learning with Local and Global Representations
- Adaptive Federated Optimization