Hierarchical Federated Learning through LAN-WAN Orchestration

Abstract

Federated learning (FL) was designed to enable mobile phones to collaboratively learn a global model without uploading their private data to a cloud server. However, exiting FL protocols has a critical communication bottleneck in a federated network coupled with privacy concerns, usually powered by a wide-area network (WAN). Such a WAN-driven FL design leads to significantly high cost and much slower model convergence. In this work, we propose an efficient FL protocol, which involves a hierarchical aggregation mechanism in the local-area network (LAN) due to its abundant bandwidth and almost negligible monetary cost than WAN. Our proposed FL can accelerate the learning process and reduce the monetary cost with frequent local aggregation in the same LAN and infrequent global aggregation on a cloud across WAN. We further design a concrete FL platform, namely LanFL, that incorporates several key techniques to handle those challenges introduced by LAN: cloud-device aggregation architecture, intra-LAN peer-to-peer (p2p) topology generation, inter-LAN bandwidth capacity heterogeneity. We evaluate LanFL on 2 typical Non-IID datasets, which reveals that LanFL can significantly accelerate FL training (1.5x-6.0x), save WAN traffic (18.3x-75.6x), and reduce monetary cost (3.8x-27.2x) while preserving the model accuracy.

References in corpus (19)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Federated Learning for Mobile Keyboard Prediction
• Towards Federated Learning at Scale: System Design
• Horovod: fast and easy distributed deep learning in TensorFlow
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Federated Optimization in Heterogeneous Networks
• A generic framework for privacy preserving deep learning
• Local SGD Converges Fast and Communicates Little
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Communication Compression for Decentralized Training
• D${}^2$: Decentralized Training over Decentralized Data
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Peer-to-peer Federated Learning on Graphs
• Federated Learning Of Out-Of-Vocabulary Words
• Gossip Dual Averaging for Decentralized Optimization of Pairwise Functions
• Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data

Cited by in corpus (4)

• From Distributed Machine Learning to Federated Learning: A Survey
• Analysis and Optimal Edge Assignment For Hierarchical Federated Learning on Non-IID Data
• DYNAMITE: Dynamic Interplay of Mini-Batch Size and Aggregation Frequency for Federated Learning with Static and Streaming Dataset
• Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data