A General Distributed Dual Coordinate Optimization Framework for Regularized Loss Minimization

Abstract

In modern large-scale machine learning applications, the training data are often partitioned and stored on multiple machines. It is customary to employ the "data parallelism" approach, where the aggregated training loss is minimized without moving data across machines. In this paper, we introduce a novel distributed dual formulation for regularized loss minimization problems that can directly handle data parallelism in the distributed setting. This formulation allows us to systematically derive dual coordinate optimization procedures, which we refer to as Distributed Alternating Dual Maximization (DADM). The framework extends earlier studies described in (Boyd et al., 2011; Ma et al., 2015a; Jaggi et al., 2014; Yang, 2013) and has rigorous theoretical analyses. Moreover with the help of the new formulation, we develop the accelerated version of DADM (Acc-DADM) by generalizing the acceleration technique from (Shalev-Shwartz and Zhang, 2014) to the distributed setting. We also provide theoretical results for the proposed accelerated version and the new result improves previous ones (Yang, 2013; Ma et al., 2015a) whose runtimes grow linearly on the condition number. Our empirical studies validate our theory and show that our accelerated approach significantly improves the previous state-of-the-art distributed dual coordinate optimization algorithms.

Cited by in corpus (8)

• On the Convergence of FedAvg on Non-IID Data
• The Heterogeneous Ensembles of Standard Classification Algorithms (HESCA): the Whole is Greater than the Sum of its Parts
• GIANT: Globally Improved Approximate Newton Method for Distributed Optimization
• Solving Empirical Risk Minimization in the Current Matrix Multiplication Time
• L1-Regularized Distributed Optimization: A Communication-Efficient Primal-Dual Framework
• The Scalability for Parallel Machine Learning Training Algorithm: Dataset Matters
• Distributed Block-diagonal Approximation Methods for Regularized Empirical Risk Minimization
• A Distributed Quasi-Newton Algorithm for Primal and Dual Regularized Empirical Risk Minimization