873 citations · 1.3k across the 35 of their papers we have counts for
37 papers
Momentum Benefits Non-IID Federated Learning Simply and Provably
Ziheng Cheng, Xinmeng Huang, Pengfei Wu +1
Federated learning is a powerful paradigm for large-scale machine learning, but it faces significant challenges due to unreliable network connections, slow communication, and subst…
DSGD-CECA: Decentralized SGD with Communication-Optimal Exact Consensus Algorithm
Lisang Ding, Kexin Jin, Bicheng Ying +2
Decentralized Stochastic Gradient Descent (SGD) is an emerging neural network training approach that enables multiple agents to train a model collaboratively and simultaneously. Ra…
Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?
Yutong He, Xinmeng Huang, Kun Yuan
Communication compression is a common technique in distributed optimization that can alleviate communication overhead by transmitting compressed gradients and model parameters. How…
Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression
Yutong He, Xinmeng Huang, Yiming Chen +2
Communication compression is an essential strategy for alleviating communication overhead by reducing the volume of information exchanged between computing nodes in large-scale dis…
An Enhanced Gradient-Tracking Bound for Distributed Online Stochastic Convex Optimization
Sulaiman A. Alghunaim, Kun Yuan
Gradient-tracking (GT) based decentralized methods have emerged as an effective and viable alternative method to decentralized (stochastic) gradient descent (DSGD) when solving dis…
Optimal Complexity in Non-Convex Decentralized Learning over Time-Varying Networks
Xinmeng Huang, Kun Yuan
Decentralized optimization with time-varying networks is an emerging paradigm in machine learning. It saves remarkable communication overhead in large-scale deep training and is mo…
Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimization
Kun Yuan, Xinmeng Huang, Yiming Chen +3
Decentralized optimization is effective to save communication in large-scale machine learning. Although numerous algorithms have been proposed with theoretical guarantees and empir…
On the Performance of Gradient Tracking with Local Updates
Edward Duc Hien Nguyen, Sulaiman A. Alghunaim, Kun Yuan +1
We study the decentralized optimization problem where a network of agents seeks to minimize the average of a set of heterogeneous non-convex cost functions distributedly. State…
Communication-Efficient Topologies for Decentralized Learning with Consensus Rate
Zhuoqing Song, Weijian Li, Kexin Jin +4
Decentralized optimization is an emerging paradigm in distributed learning in which agents achieve network-wide solutions by peer-to-peer communication without the central server.…
Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression
Xinmeng Huang, Yiming Chen, Wotao Yin +1
Recent advances in distributed optimization and learning have shown that communication compression is one of the most effective means of reducing communication. While there have be…
Heavy-Tail Phenomenon in Decentralized SGD
Mert Gurbuzbalaban, Yuanhan Hu, Umut Simsekli +2
Recent theoretical studies have shown that heavy-tails can emerge in stochastic optimization due to `multiplicative noise', even under surprisingly simple settings, such as linear…
Exponential Graph is Provably Efficient for Decentralized Deep Training
Bicheng Ying, Kun Yuan, Yiming Chen +3
Decentralized SGD is an emerging training method for deep learning known for its much less (thus faster) communication per iteration, which relaxes the averaging step in parallel S…
Decentralized Composite Optimization with Compression
Yao Li, Xiaorui Liu, Jiliang Tang +2
Decentralized optimization and communication compression have exhibited their great potential in accelerating distributed machine learning by mitigating the communication bottlenec…
Accelerating Gossip SGD with Periodic Global Averaging
Yiming Chen, Kun Yuan, Yingya Zhang +3
Communication overhead hinders the scalability of large-scale distributed training. Gossip SGD, where each node averages only with its neighbors, is more communication-efficient th…
Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD
Kun Yuan, Sulaiman A. Alghunaim, Xinmeng Huang
We consider the decentralized stochastic optimization problems, where a network of nodes, each owning a local cost function, cooperate to find a minimizer of the globally-avera…
DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training
Kun Yuan, Yiming Chen, Xinmeng Huang +4
The scale of deep learning nowadays calls for efficient distributed training algorithms. Decentralized momentum SGD (DmSGD), in which each node averages only with its neighbors, is…
Improved Analysis and Rates for Variance Reduction under Without-replacement Sampling Orders
Xinmeng Huang, Kun Yuan, Xianghui Mao +1
When applying a stochastic algorithm, one must choose an order to draw samples. The practical choices are without-replacement sampling orders, which are empirically faster and more…
Can Primal Methods Outperform Primal-dual Methods in Decentralized Dynamic Optimization?
Kun Yuan, Wei Xu, Qing Ling
In this paper, we consider the decentralized dynamic optimization problem defined over a multi-agent network. Each agent possesses a time-varying local objective function, and all…
Decentralized Proximal Gradient Algorithms with Linear Convergence Rates
Sulaiman A. Alghunaim, Ernest K. Ryu, Kun Yuan +1
This work studies a class of non-smooth decentralized multi-agent optimization problems where the agents aim at minimizing a sum of local strongly-convex smooth components plus a c…
ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems
Ernest K. Ryu, Kun Yuan, Wotao Yin
Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poo…
A Linearly Convergent Proximal Gradient Algorithm for Decentralized Optimization
Sulaiman A. Alghunaim, Kun Yuan, Ali H. Sayed
Decentralized optimization is a powerful paradigm that finds applications in engineering and learning design. This work studies decentralized composite optimization problems with n…
On the Influence of Bias-Correction on Distributed Stochastic Optimization
Kun Yuan, Sulaiman A. Alghunaim, Bicheng Ying +1
Various bias-correction methods such as EXTRA, gradient tracking methods, and exact diffusion have been proposed recently to solve distributed {\em deterministic} optimization prob…
Dynamic Average Diffusion with randomized Coordinate Updates
Bicheng Ying, Kun Yuan, Ali H. Sayed
This work derives and analyzes an online learning strategy for tracking the average of time-varying distributed signals by relying on randomized coordinate-descent updates. During…
Multi-Agent Fully Decentralized Value Function Learning with Linear Convergence Rates
Lucas Cassano, Kun Yuan, Ali H. Sayed
This work develops a fully decentralized multi-agent algorithm for policy evaluation. The proposed scheme can be applied to two distinct scenarios. In the first scenario, a collect…