Gradient tracking and variance reduction for decentralized optimization and machine learning
arXiv:2002.05373
Abstract
Decentralized methods to solve finite-sum minimization problems are important in many signal processing and machine learning tasks where the data is distributed over a network of nodes and raw data sharing is not permitted due to privacy and/or resource constraints. In this article, we review decentralized stochastic first-order methods and provide a unified algorithmic framework that combines variance-reduction with gradient tracking to achieve both robust performance and fast convergence. We provide explicit theoretical guarantees of the corresponding methods when the objective functions are smooth and strongly-convex, and show their applicability to non-convex problems via numerical experiments. Throughout the article, we provide intuitive illustrations of the main technical ideas by casting appropriate tradeoffs and comparisons among the methods of interest and by highlighting applications to decentralized training of machine learning models.
accepted for publication, IEEE Signal Processing Magazine
References in corpus (12)
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Diffusion Adaptation Strategies for Distributed Optimization and Learning over Networks
- Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
- DSA: Decentralized Double Stochastic Averaging Gradient Algorithm
- An Exact Quantized Decentralized Gradient Descent Algorithm
- Stochastic Gradient Push for Distributed Deep Learning
- Variance-Reduced Decentralized Stochastic Optimization with Accelerated Convergence
- A Linearly Convergent Proximal Gradient Algorithm for Decentralized Optimization
- A Sharp Estimate on the Transient Time of Distributed Stochastic Gradient Descent
- Asynchronous Accelerated Proximal Stochastic Gradient for Strongly Convex Distributed Finite Sums
- Variance-Reduced Stochastic Learning under Random Reshuffling
- Communication-Efficient Distributed Optimization in Networks with Gradient Tracking and Variance Reduction