Boosting Asynchronous Decentralized Learning with Model Fragmentation

Abstract

Decentralized learning (DL) is an emerging technique that allows nodes on the web to collaboratively train machine learning models without sharing raw data. Dealing with stragglers, i.e., nodes with slower compute or communication than others, is a key challenge in DL. We present DivShare, a novel asynchronous DL algorithm that achieves fast model convergence in the presence of communication stragglers. DivShare achieves this by having nodes fragment their models into parameter subsets and send, in parallel to computation, each subset to a random sample of other nodes instead of sequentially exchanging full models. The transfer of smaller fragments allows more efficient usage of the collective bandwidth and enables nodes with slow network links to quickly contribute with at least some of their model parameters. By theoretically proving the convergence of DivShare, we provide, to the best of our knowledge, the first formal proof of convergence for a DL algorithm that accounts for the effects of asynchronous communication with delays. We experimentally evaluate DivShare against two state-of-the-art DL baselines, AD-PSGD and Swift, and with two standard datasets, CIFAR-10 and MovieLens. We find that DivShare with communication stragglers lowers time-to-accuracy by up to 3.9x compared to AD-PSGD on the CIFAR-10 dataset. Compared to baselines, DivShare also achieves up to 19.4% better accuracy and 9.5% lower test loss on the CIFAR-10 and MovieLens datasets, respectively.

Accepted to appear in the Proceedings of the ACM Web Conference 2025 (WWW '25)

References in corpus (23)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges
• Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Gossip Learning with Linear Models on Fully Distributed Data
• Stochastic Gradient Push for Distributed Deep Learning
• Resource-Efficient Federated Learning
• Asynchronous Decentralized Parallel Stochastic Gradient Descent
• Asynchronous Gradient-Push
• FedScale: Benchmarking Model and System Performance of Federated Learning at Scale
• Federated Learning with Buffered Asynchronous Aggregation
• Decentralized Deep Learning with Arbitrary Communication Compression
• Papaya: Practical, Private, and Scalable Federated Learning
• Asynchronous Decentralized SGD with Quantized and Local Updates
• Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning
• Decentralized Learning Made Easy with DecentralizePy
• Epidemic Learning: Boosting Decentralized Learning with Randomized Communication
• TEE-based decentralized recommender systems: The raw data sharing redemption
• Get More for Less in Decentralized Learning Systems
• SWIFT: Rapid Decentralized Federated Learning via Wait-Free Model Communication
• Communication-Efficient Topologies for Decentralized Learning with $O(1)$ Consensus Rate
• Asynchronous SGD on Graphs: a Unified Framework for Asynchronous Decentralized and Federated Optimization