An introduction to decentralized stochastic optimization with gradient tracking
arXiv:1907.09648
Abstract
Decentralized solutions to finite-sum minimization are of significant importance in many signal processing, control, and machine learning applications. In such settings, the data is distributed over a network of arbitrarily-connected nodes and raw data sharing is prohibitive often due to communication or privacy constraints. In this article, we review decentralized stochastic first-order optimization methods and illustrate some recent improvements based on gradient tracking and variance reduction, focusing particularly on smooth and strongly-convex objective functions. We provide intuitive illustrations of the main technical ideas as well as applications of the algorithms in the context of decentralized training of machine learning models.
References in corpus (14)
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Diffusion Adaptation Strategies for Distributed Optimization and Learning over Networks
- Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
- Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
- DSA: Decentralized Double Stochastic Averaging Gradient Algorithm
- An Exact Quantized Decentralized Gradient Descent Algorithm
- Stochastic Gradient Push for Distributed Deep Learning
- On the Influence of Bias-Correction on Distributed Stochastic Optimization
- A Simple Practical Accelerated Method for Finite Sums
- A Linearly Convergent Proximal Gradient Algorithm for Decentralized Optimization
- Asynchronous Accelerated Proximal Stochastic Gradient for Strongly Convex Distributed Finite Sums
- Variance-Reduced Stochastic Learning under Random Reshuffling
- Variance-Reduced Decentralized Stochastic Optimization with Gradient Tracking--Part I: GT-SAGA
- Distributed heavy-ball: A generalization and acceleration of first-order methods with gradient tracking
Cited by in corpus (5)
- Can Primal Methods Outperform Primal-dual Methods in Decentralized Dynamic Optimization?
- Federated Learning over Wireless Device-to-Device Networks: Algorithms and Convergence Analysis
- Channel-driven Decentralized Bayesian Federated Learning for Trustworthy Decision Making in D2D Networks
- Decentralized Federated Learning via SGD over Wireless D2D Networks
- Communication-efficient Decentralized Machine Learning over Heterogeneous Networks