CoCoA: A General Framework for Communication-Efficient Distributed Optimization

Abstract

The scale of modern datasets necessitates the development of efficient distributed optimization methods for machine learning. We present a general-purpose framework for distributed computing environments, CoCoA, that has an efficient communication scheme and is applicable to a wide variety of problems in machine learning and signal processing. We extend the framework to cover general non-strongly-convex regularizers, including L1-regularized problems like lasso, sparse logistic regression, and elastic net regularization, and show how earlier work can be derived as a special case. We provide convergence guarantees for the class of convex regularized loss minimization objectives, leveraging a novel approach in handling non-strongly-convex regularizers and non-smooth loss functions. The resulting framework has markedly improved performance over state-of-the-art methods, as we illustrate with an extensive set of experiments on real distributed datasets.

References in corpus (1)

• Distributed Optimization with Arbitrary Local Solvers

Cited by in corpus (21)

• On the Convergence of FedAvg on Non-IID Data
• FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
• Robust Federated Learning in a Heterogeneous Environment
• Multi-Stage Hybrid Federated Learning over Large-Scale D2D-Enabled Fog Networks
• Federated Multi-Task Learning
• Towards Demystifying Serverless Machine Learning Training
• Central Server Free Federated Learning over Single-sided Trust Social Networks
• Least Squares Approximation for a Distributed System
• GIANT: Globally Improved Approximate Newton Method for Distributed Optimization
• SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networks
• Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data
• Global linear convergence of Newton's method without strong-convexity or Lipschitz gradients
• Communication trade-offs for synchronized distributed SGD with large step size
• Distributed linear regression by averaging
• DAve-QN: A Distributed Averaged Quasi-Newton Method with Local Superlinear Convergence Rate
• Understanding and Optimizing the Performance of Distributed Machine Learning Applications on Apache Spark
• Snap ML: A Hierarchical Framework for Machine Learning
• A Primal-Dual Algorithm for Hybrid Federated Learning
• LocalNewton: Reducing Communication Bottleneck for Distributed Learning
• Optimal Data Splitting in Distributed Optimization for Machine Learning
• Local Methods with Adaptivity via Scaling