Accelerating Federated Learning in Heterogeneous Data and Computational Environments
arXiv:2008.11281
Abstract
There are situations where data relevant to a machine learning problem are distributed among multiple locations that cannot share the data due to regulatory, competitiveness, or privacy reasons. For example, data present in users' cellphones, manufacturing data of companies in a given industrial sector, or medical records located at different hospitals. Moreover, participating sites often have different data distributions and computational capabilities. Federated Learning provides an approach to learn a joint model over all the available data in these environments. In this paper, we introduce a novel distributed validation weighting scheme (DVW), which evaluates the performance of a learner in the federation against a distributed validation set. Each learner reserves a small portion (e.g., 5%) of its local training examples as a validation dataset and allows other learners models to be evaluated against it. We empirically show that DVW results in better performance compared to established methods, such as FedAvg, both under synchronous and asynchronous communication protocols in data and computationally heterogeneous environments.
References in corpus (11)
- Deep Learning with Differential Privacy
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning: Strategies for Improving Communication Efficiency
- Towards Federated Learning at Scale: System Design
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
- Revisiting Distributed Synchronous SGD
- Federated Optimization in Heterogeneous Networks
- Asynchronous Federated Optimization
- LEAF: A Benchmark for Federated Settings
- Adaptive Federated Optimization
- Blind Justice: Fairness with Encrypted Sensitive Attributes