Distributed Learning in the Non-Convex World: From Batch to Streaming Data, and Beyond

Abstract

Distributed learning has become a critical enabler of the massively connected world envisioned by many. This article discusses four key elements of scalable distributed processing and real-time intelligence --- problems, data, communication and computation. Our aim is to provide a fresh and unique perspective about how these elements should work together in an effective and coherent manner. In particular, we {provide a selective review} about the recent techniques developed for optimizing non-convex models (i.e., problem classes), processing batch and streaming data (i.e., data types), over the networks in a distributed manner (i.e., communication and computation paradigm). We describe the intuitions and connections behind a core set of popular distributed algorithms, emphasizing how to trade off between computation and communication costs. Practical issues and future research directions will also be discussed.

Submitted to IEEE Signal Processing Magazine Special Issue on Distributed, Streaming Machine Learning; THC, MH, HTW contributed equally

References in corpus (4)

• Multi-Agent Distributed Optimization via Inexact Consensus ADMM
• Convex Optimization for Big Data
• ByRDiE: Byzantine-resilient distributed coordinate descent for decentralized learning
• Distributed Non-Convex First-Order Optimization and Information Processing: Lower Complexity Bounds and Rate Optimal Algorithms

Cited by in corpus (7)

• Non-convex Min-Max Optimization: Applications, Challenges, and Recent Theoretical Advances
• Rethinking Bayesian Learning for Data Analysis: The Art of Prior and Inference in Sparsity-Aware Modeling
• DiNNO: Distributed Neural Network Optimization for Multi-Robot Collaborative Learning
• Distributed Stochastic Consensus Optimization with Momentum for Nonconvex Nonsmooth Problems
• Internal Model-Based Online Optimization
• Consensus-Based Optimization for Saddle Point Problems
• Federated Learning over Wireless Device-to-Device Networks: Algorithms and Convergence Analysis