TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture

Abstract

Federated learning has emerged as a new paradigm of collaborative machine learning; however, many prior studies have used global aggregation along a star topology without much consideration of the communication scalability or the diurnal property relied on clients' local time variety. In contrast, ring architecture can resolve the scalability issue and even satisfy the diurnal property by iterating nodes without an aggregation. Nevertheless, such ring-based algorithms can inherently suffer from the high-variance problem. To this end, we propose a novel algorithm called TornadoAggregate that improves both accuracy and scalability by facilitating the ring architecture. In particular, to improve the accuracy, we reformulate the loss minimization into a variance reduction problem and establish three principles to reduce variance: Ring-Aware Grouping, Small Ring, and Ring Chaining. Experimental results show that TornadoAggregate improved the test accuracy by up to 26.7% and achieved near-linear scalability.

References in corpus (18)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Towards Federated Learning at Scale: System Design
• FedMD: Heterogenous Federated Learning via Model Distillation
• Federated Optimization in Heterogeneous Networks
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Don't Use Large Mini-Batches, Use Local SGD
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Robust Federated Learning in a Heterogeneous Environment
• Overcoming Forgetting in Federated Learning on Non-IID Data
• Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training
• MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling
• Federated Continual Learning with Weighted Inter-client Transfer
• Hybrid-FL for Wireless Networks: Cooperative Learning Mechanism Using Non-IID Data
• Distributed Optimization over Block-Cyclic Data

Cited by in corpus (3)

• FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measure
• Flexible Clustered Federated Learning for Client-Level Data Distribution Shift
• Basil: A Fast and Byzantine-Resilient Approach for Decentralized Training