Analysis and Optimal Edge Assignment For Hierarchical Federated Learning on Non-IID Data

Abstract

Distributed learning algorithms aim to leverage distributed and diverse data stored at users' devices to learn a global phenomena by performing training amongst participating devices and periodically aggregating their local models' parameters into a global model. Federated learning is a promising paradigm that allows for extending local training among the participant devices before aggregating the parameters, offering better communication efficiency. However, in the cases where the participants' data are strongly skewed (i.e., non-IID), the local models can overfit local data, leading to low performing global model. In this paper, we first show that a major cause of the performance drop is the weighted distance between the distribution over classes on users' devices and the global distribution. Then, to face this challenge, we leverage the edge computing paradigm to design a hierarchical learning system that performs Federated Gradient Descent on the user-edge layer and Federated Averaging on the edge-cloud layer. In this hierarchical architecture, we formalize and optimize this user-edge assignment problem such that edge-level data distributions turn to be similar (i.e., close to IID), which enhances the Federated Averaging performance. Our experiments on multiple real-world datasets show that the proposed optimized assignment is tractable and leads to faster convergence of models towards a better accuracy value.

References in corpus (12)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Challenges, Methods, and Future Directions
• Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
• In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communication by Federated Learning
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Revisiting Distributed Synchronous SGD
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• Practical Secure Aggregation for Federated Learning on User-Held Data
• Accelerating Federated Learning over Reliability-Agnostic Clients in Mobile Edge Computing Systems
• Reconfigurable Wireless Networks
• Hierarchical Federated Learning through LAN-WAN Orchestration

Cited by in corpus (7)

• From Distributed Machine Learning to Federated Learning: A Survey
• Pervasive AI for IoT applications: A Survey on Resource-efficient Distributed Artificial Intelligence
• Topology-aware Federated Learning in Edge Computing: A Comprehensive Survey
• Federated Learning Survey: A Multi-Level Taxonomy of Aggregation Techniques, Experimental Insights, and Future Frontiers
• Optimal Resource Management for Hierarchical Federated Learning over HetNets with Wireless Energy Transfer
• Reinforcement Learning for Intelligent Healthcare Systems: A Comprehensive Survey
• Energy-Efficient Multi-Orchestrator Mobile Edge Learning