Optimal Gradient Tracking for Decentralized Optimization

Abstract

In this paper, we focus on solving the decentralized optimization problem of minimizing the sum of $n$ objective functions over a multi-agent network. The agents are embedded in an undirected graph where they can only send/receive information directly to/from their immediate neighbors. Assuming smooth and strongly convex objective functions, we propose an Optimal Gradient Tracking (OGT) method that achieves the optimal gradient computation complexity $O\left(\sqrt{\kappa }\log \frac{1}{\epsilon }\right)$ and the optimal communication complexity $O\left(\sqrt{\frac{\kappa }{\theta }}\log \frac{1}{\epsilon }\right)$ simultaneously, where $\kappa$ and $\frac{1}{\theta }$ denote the condition numbers related to the objective functions and the communication graph, respectively. To our knowledge, OGT is the first single-loop decentralized gradient-type method that is optimal in both gradient computation and communication complexities. The development of OGT involves two building blocks which are also of independent interest. The first one is another new decentralized gradient tracking method termed "Snapshot" Gradient Tracking (SS-GT), which achieves the gradient computation and communication complexities of $O\left(\sqrt{\kappa }\log \frac{1}{\epsilon }\right)$ and $O\left(\frac{\sqrt{\kappa }}{\theta }\log \frac{1}{\epsilon }\right)$, respectively. SS-GT can be potentially extended to more general settings compared to OGT. The second one is a technique termed Loopless Chebyshev Acceleration (LCA) which can be implemented "looplessly" but achieve similar effect with adding multiple inner loops of Chebyshev acceleration in the algorithms. In addition to SS-GT, this LCA technique can accelerate many other gradient tracking based methods with respect to the graph condition number $\frac{1}{\theta }$.

Mathematical Programming, in press

References in corpus (12)

• Multi-Agent Distributed Optimization via Inexact Consensus ADMM
• A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates
• Optimal algorithms for smooth and strongly convex distributed optimization in networks
• Distributed Recursive Least-Squares: Stability and Performance Analysis
• Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis
• Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization
• Multi-consensus Decentralized Accelerated Gradient Descent
• Efficient Information Aggregation Strategies for Distributed Control and Signal Processing
• Robust Distributed Accelerated Stochastic Gradient Methods for Multi-Agent Networks
• ADOM: Accelerated Decentralized Optimization Method for Time-Varying Networks
• Accelerated Gradient Tracking over Time-varying Graphs for Decentralized Optimization
• Provably Accelerated Decentralized Gradient Method Over Unbalanced Directed Graphs

Cited by in corpus (1)

• Decentralized Distributed Optimization for Saddle Point Problems