873 citations · 1.3k across the 35 of their papers we have counts for
6 papers · 1 filter
Exponential Graph is Provably Efficient for Decentralized Deep Training
Bicheng Ying, Kun Yuan, Yiming Chen +3
Decentralized SGD is an emerging training method for deep learning known for its much less (thus faster) communication per iteration, which relaxes the averaging step in parallel S…
Decentralized Composite Optimization with Compression
Yao Li, Xiaorui Liu, Jiliang Tang +2
Decentralized optimization and communication compression have exhibited their great potential in accelerating distributed machine learning by mitigating the communication bottlenec…
Accelerating Gossip SGD with Periodic Global Averaging
Yiming Chen, Kun Yuan, Yingya Zhang +3
Communication overhead hinders the scalability of large-scale distributed training. Gossip SGD, where each node averages only with its neighbors, is more communication-efficient th…
Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD
Kun Yuan, Sulaiman A. Alghunaim, Xinmeng Huang
We consider the decentralized stochastic optimization problems, where a network of nodes, each owning a local cost function, cooperate to find a minimizer of the globally-avera…
DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training
Kun Yuan, Yiming Chen, Xinmeng Huang +4
The scale of deep learning nowadays calls for efficient distributed training algorithms. Decentralized momentum SGD (DmSGD), in which each node averages only with its neighbors, is…
Improved Analysis and Rates for Variance Reduction under Without-replacement Sampling Orders
Xinmeng Huang, Kun Yuan, Xianghui Mao +1
When applying a stochastic algorithm, one must choose an order to draw samples. The practical choices are without-replacement sampling orders, which are empirically faster and more…