Cost-Effective Federated Learning Design
arXiv:2012.08336
Abstract
Federated learning (FL) is a distributed learning paradigm that enables a large number of devices to collaboratively learn a model without sharing their raw data. Despite its practical efficiency and effectiveness, the iterative on-device learning process incurs a considerable cost in terms of learning time and energy consumption, which depends crucially on the number of selected clients and the number of local iterations in each training round. In this paper, we analyze how to design adaptive FL that optimally chooses these essential control variables to minimize the total cost while ensuring convergence. Theoretically, we analytically establish the relationship between the total cost and the control variables with the convergence upper bound. To efficiently solve the cost minimization problem, we develop a low-cost sampling-based algorithm to learn the convergence related unknown parameters. We derive important solution properties that effectively identify the design principles for different metric preferences. Practically, we evaluate our theoretical results both in a simulated environment and on a hardware prototype. Experimental evidence verifies our derived properties and demonstrates that our proposed solution achieves near-optimal performance for various datasets, different machine learning models, and heterogeneous system settings.
arXiv commentAccepted in IEEE INFOCOM 2021
References in corpus (8)
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning: Strategies for Improving Communication Efficiency
- Federated Optimization: Distributed Machine Learning for On-Device Intelligence
- Practical Secure Aggregation for Federated Learning on User-Held Data
- First Analysis of Local GD on Heterogeneous Data
- Convergence Time Optimization for Federated Learning over Wireless Networks
- Network-Aware Optimization of Distributed Learning for Fog Computing