Adaptive Federated Learning in Resource Constrained Edge Computing Systems

Abstract

Emerging technologies and applications including Internet of Things (IoT), social networking, and crowd-sourcing generate large amounts of data at the network edge. Machine learning models are often built from the collected data, to enable the detection, classification, and prediction of future events. Due to bandwidth, storage, and privacy concerns, it is often impractical to send all the data to a centralized location. In this paper, we consider the problem of learning model parameters from data distributed across multiple edge nodes, without sending raw data to a centralized place. Our focus is on a generic class of machine learning models that are trained using gradient-descent based approaches. We analyze the convergence bound of distributed gradient descent from a theoretical point of view, based on which we propose a control algorithm that determines the best trade-off between local update and global parameter aggregation to minimize the loss function under a given resource budget. The performance of the proposed algorithm is evaluated via extensive experiments with real datasets, both on a networked prototype system and in a larger-scale simulated environment. The experimentation results show that our proposed approach performs near to the optimum with various machine learning models and different data distributions.

arXiv commentThis version (excluding appendices) has been accepted for publication in the IEEE Journal on Selected Areas in Communications

References in corpus (12)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Revisiting Distributed Synchronous SGD
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Practical Secure Aggregation for Federated Learning on User-Held Data
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Asynchronous Stochastic Gradient Descent with Delay Compensation
• Communication Complexity of Distributed Convex Learning and Optimization
• Asynchronous Decentralized Parallel Stochastic Gradient Descent

Cited by in corpus (7)

• Federated Machine Learning: Concept and Applications
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• First Analysis of Local GD on Heterogeneous Data
• Tighter Theory for Local SGD on Identical and Heterogeneous Data
• Federated AI for building AI Solutions across Multiple Agencies
• Gradient Descent with Compressed Iterates
• Blockchain Assisted Decentralized Federated Learning (BLADE-FL) with Lazy Clients