Scaling-up Distributed Processing of Data Streams for Machine Learning
arXiv:2005.08854 · doi:10.1109/JPROC.2020.3021381
Abstract
Emerging applications of machine learning in numerous areas involve continuous gathering of and learning from streams of data. Real-time incorporation of streaming data into the learned models is essential for improved inference in these applications. Further, these applications often involve data that are either inherently gathered at geographically distributed entities or that are intentionally distributed across multiple machines for memory, computational, and/or privacy reasons. Training of models in this distributed, streaming setting requires solving stochastic optimization problems in a collaborative manner over communication links between the physical entities. When the streaming data rate is high compared to the processing capabilities of compute nodes and/or the rate of the communications links, this poses a challenging question: how can one best leverage the incoming data for distributed training under constraints on computing capabilities and/or communications rate? A large body of research has emerged in recent decades to tackle this and related problems. This paper reviews recently developed methods that focus on large-scale distributed stochastic optimization in the compute- and bandwidth-limited regime, with an emphasis on convergence analysis that explicitly accounts for the mismatch between computation, communication and streaming rates. In particular, it focuses on methods that solve: (i) distributed stochastic convex problems, and (ii) distributed principal component analysis, which is a nonconvex problem with geometric structure that permits global convergence. For such methods, the paper discusses recent advances in terms of distributed algorithmic designs when faced with high-rate streaming data. Further, it reviews guarantees underlying these methods, which show there exist regimes in which systems can learn from distributed, streaming data at order-optimal rates.
arXiv comment45 pages, 9 figures; preprint of a journal paper published in Proceedings of the IEEE (Special Issue on Optimization for Data-driven Learning and Control)
References in corpus (16)
- Federated Learning: Challenges, Methods, and Future Directions
- Federated Optimization: Distributed Machine Learning for On-Device Intelligence
- Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling
- On the Linear Convergence of the ADMM in Decentralized Consensus Optimization
- Gossip Algorithms for Distributed Signal Processing
- Diffusion Adaptation Strategies for Distributed Optimization and Learning over Networks
- A linear algorithm for optimization over directed graphs with geometric convergence
- Variance Reduction for Faster Non-Convex Optimization
- DQM: Decentralized Quadratically Approximated Alternating Direction Method of Multipliers
- Decentralized Frank-Wolfe Algorithm for Convex and Non-convex Problems
- Adversary-resilient Distributed and Decentralized Statistical Inference and Machine Learning: An Overview of Recent Advances Under the Byzantine Threat Model
- Riemannian stochastic variance reduced gradient algorithm with retraction and vector transport
- Natasha 2: Faster Non-Convex Optimization Than SGD
- Natasha: Faster Non-Convex Stochastic Optimization Via Strongly Non-Convex Parameter
- Improved SVRG for Non-Strongly-Convex or Sum-of-Non-Convex Objectives
- Stochastic Optimization from Distributed, Streaming Data in Rate-limited Networks