Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementation

Abstract

The conventional federated learning (FedL) architecture distributes machine learning (ML) across worker devices by having them train local models that are periodically aggregated by a server. FedL ignores two important characteristics of contemporary wireless networks, however: (i) the network may contain heterogeneous communication/computation resources, while (ii) there may be significant overlaps in devices' local data distributions. In this work, we develop a novel optimization methodology that jointly accounts for these factors via intelligent device sampling complemented by device-to-device (D2D) offloading. Our optimization aims to select the best combination of sampled nodes and data offloading configuration to maximize FedL training accuracy subject to realistic constraints on the network topology and device capabilities. Theoretical analysis of the D2D offloading subproblem leads to new FedL convergence bounds and an efficient sequential convex optimizer. Using this result, we develop a sampling methodology based on graph convolutional networks (GCNs) which learns the relationship between network attributes, sampled nodes, and resulting offloading that maximizes FedL accuracy. Through evaluation on real-world datasets and network measurements from our IoT testbed, we find that our methodology while sampling less than 5% of all devices outperforms conventional FedL substantially both in terms of trained model accuracy and required resource utilization.

This paper is accepted for publication in the proceedings of 2021 IEEE International Conference on Computer Communications (INFOCOM)

References in corpus (4)

• A Comprehensive Survey on Graph Neural Networks
• Federated Learning: Challenges, Methods, and Future Directions
• Federated Learning: Strategies for Improving Communication Efficiency
• Joint Optimization of Signal Design and Resource Allocation in Wireless D2D Edge Computing

Cited by in corpus (8)

• The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retraining
• Multi-Stage Hybrid Federated Learning over Large-Scale D2D-Enabled Fog Networks
• AnycostFL: Efficient On-Demand Federated Learning over Heterogeneous Edge Devices
• Federated Few-Shot Learning for Mobile NLP
• Multi-Edge Server-Assisted Dynamic Federated Learning with an Optimized Floating Aggregation Point
• Accelerating Vertical Federated Learning
• Towards Practical Few-shot Federated NLP
• UAV-assisted Online Machine Learning over Multi-Tiered Networks: A Hierarchical Nested Personalized Federated Learning Approach