Multi-Stage Hybrid Federated Learning over Large-Scale D2D-Enabled Fog Networks

Abstract

Federated learning has generated significant interest, with nearly all works focused on a "star" topology where nodes/devices are each connected to a central server. We migrate away from this architecture and extend it through the network dimension to the case where there are multiple layers of nodes between the end devices and the server. Specifically, we develop multi-stage hybrid federated learning (MH-FL), a hybrid of intra- and inter-layer model learning that considers the network as a multi-layer cluster-based structure. MH-FL considers the topology structures among the nodes in the clusters, including local networks formed via device-to-device (D2D) communications, and presumes a semi-decentralized architecture for federated learning. It orchestrates the devices at different network layers in a collaborative/cooperative manner (i.e., using D2D interactions) to form local consensus on the model parameters and combines it with multi-stage parameter relaying between layers of the tree-shaped hierarchy. We derive the upper bound of convergence for MH-FL with respect to parameters of the network topology (e.g., the spectral radius) and the learning algorithm (e.g., the number of D2D rounds in different clusters). We obtain a set of policies for the D2D rounds at different clusters to guarantee either a finite optimality gap or convergence to the global optimum. We then develop a distributed control algorithm for MH-FL to tune the D2D rounds in each cluster over time to meet specific convergence criteria. Our experiments on real-world datasets verify our analytical results and demonstrate the advantages of MH-FL in terms of resource utilization metrics.

This paper is accepted for publication in IEEE/ACM Transactions on Networking

References in corpus (22)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Differentially Private Federated Learning: A Client Level Perspective
• Multi-Agent Distributed Optimization via Inexact Consensus ADMM
• Federated Learning with Cooperating Devices: A Consensus Approach for Massive IoT Networks
• Asynchronous Federated Optimization
• Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption
• Distributed Constrained Optimization by Consensus-Based Primal-Dual Perturbation Method
• Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocation
• FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
• Hybrid Deterministic-Stochastic Methods for Data Fitting
• Distributed Coordinate Descent Method for Learning with Big Data
• Decentralized Federated Learning: A Segmented Gossip Approach
• Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementation
• A Joint Learning and Communications Framework for Federated Learning over Wireless Networks
• CoCoA: A General Framework for Communication-Efficient Distributed Optimization
• Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learning
• GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning
• Energy Efficient Power and Channel Allocation in Underlay Device to Multi Device Communications
• Wide Compression: Tensor Ring Nets
• Federated Learning in the Sky: Joint Power Allocation and Scheduling with UAV Swarms
• Challenges and Opportunities of Future Rural Wireless Communications

Cited by in corpus (4)

• Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
• Multi-Edge Server-Assisted Dynamic Federated Learning with an Optimized Floating Aggregation Point
• Management of Resource at the Network Edge for Federated Learning
• UAV-assisted Online Machine Learning over Multi-Tiered Networks: A Hierarchical Nested Personalized Federated Learning Approach