${\text{A}}^2{\text{CiD}}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning

Abstract

Distributed training of Deep Learning models has been critical to many recent successes in the field. Current standard methods primarily rely on synchronous centralized algorithms which induce major communication bottlenecks and synchronization locks at scale. Decentralized asynchronous algorithms are emerging as a potential alternative but their practical applicability still lags. In order to mitigate the increase in communication cost that naturally comes with scaling the number of workers, we introduce a principled asynchronous, randomized, gossip-based optimization algorithm which works thanks to a continuous local momentum named ${\text{A}}^2{\text{CiD}}^2$. Our method allows each worker to continuously process mini-batches without stopping, and run a peer-to-peer averaging routine in parallel, reducing idle time. In addition to inducing a significant communication acceleration at no cost other than adding a local momentum variable, minimal adaptation is required to incorporate ${\text{A}}^2{\text{CiD}}^2$ to standard asynchronous approaches. Our theoretical analysis proves accelerated rates compared to previous asynchronous decentralized baselines and we empirically show that using our ${\text{A}}^2{\text{CiD}}^2$ momentum significantly decrease communication costs in poorly connected networks. In particular, we show consistent improvement on the ImageNet dataset using up to 64 asynchronous workers (A100 GPUs) and various communication network topologies.

References in corpus (17)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• Revisiting Distributed Synchronous SGD
• Large Batch Training of Convolutional Networks
• Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
• Optimal algorithms for smooth and strongly convex distributed optimization in networks
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
• Deep learning with Elastic Averaging SGD
• Near-Optimal Sparse Allreduce for Distributed Deep Learning
• Unified Optimal Analysis of the (Stochastic) Gradient Method
• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
• Decentralized Training of Foundation Models in Heterogeneous Environments
• Consensus Control for Decentralized Deep Learning
• An Accelerated Decentralized Stochastic Proximal Algorithm for Finite Sums
• BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning
• Optimal Complexity in Decentralized Training