Asymptotic Network Independence in Distributed Stochastic Optimization for Machine Learning
arXiv:1906.12345
Abstract
We provide a discussion of several recent results which, in certain scenarios, are able to overcome a barrier in distributed stochastic optimization for machine learning. Our focus is the so-called asymptotic network independence property, which is achieved whenever a distributed method executed over a network of n nodes asymptotically converges to the optimal solution at a comparable rate to a centralized method with the same computational power as the entire network. We explain this property through an example involving the training of ML models and sketch a short mathematical analysis for comparing the performance of distributed stochastic gradient descent (DSGD) with centralized stochastic gradient decent (SGD).
References in corpus (11)
- Diffusion Adaptation Strategies for Distributed Optimization and Learning over Networks
- Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
- A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates
- Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
- Stochastic Gradient Push for Distributed Deep Learning
- Adaptive Penalty-Based Distributed Stochastic Convex Optimization
- Supervised Learning Under Distributed Features
- A Sharp Estimate on the Transient Time of Distributed Stochastic Gradient Descent
- Robust Asynchronous Stochastic Gradient-Push: Asymptotically Optimal and Network-Independent Performance for Strongly Convex Functions
- Distributed Stochastic Gradient Tracking Methods
- Variance-Reduced Decentralized Stochastic Optimization with Gradient Tracking--Part I: GT-SAGA