Decentralized gradient methods: does topology matter?
arXiv:2002.12688
Abstract
Consensus-based distributed optimization methods have recently been advocated as alternatives to parameter server and ring all-reduce paradigms for large scale training of machine learning models. In this case, each worker maintains a local estimate of the optimal parameter vector and iteratively updates it by averaging the estimates obtained from its neighbors, and applying a correction on the basis of its local dataset. While theoretical results suggest that worker communication topology should have strong impact on the number of epochs needed to converge, previous experiments have shown the opposite conclusion. This paper sheds lights on this apparent contradiction and show how sparse topologies can lead to faster convergence even in the absence of communication delays.
A version of this paper is to appear at AISTATS 2020
Cited by in corpus (6)
- Federated Multi-Task Learning under a Mixture of Distributions
- Throughput-Optimal Topology Design for Cross-Silo Federated Learning
- Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
- Decentralized Stochastic Gradient Tracking for Non-convex Empirical Risk Minimization
- Consensus Control for Decentralized Deep Learning
- Asymptotic Network Independence and Step-Size for A Distributed Subgradient Method