Distributed Block-diagonal Approximation Methods for Regularized Empirical Risk Minimization

Abstract

In recent years, there is a growing need to train machine learning models on a huge volume of data. Designing efficient distributed optimization algorithms for empirical risk minimization (ERM) has therefore become an active and challenging research topic. In this paper, we propose a flexible framework for distributed ERM training through solving the dual problem, which provides a unified description and comparison of existing methods. Our approach requires only approximate solutions of the sub-problems involved in the optimization process, and is versatile to be applied on many large-scale machine learning problems including classification, regression, and structured prediction. We show that our approach enjoys global linear convergence for a broader class of problems, and achieves faster empirical performance, compared with existing works.

References in corpus (4)

• Block-Coordinate Frank-Wolfe Optimization for Structural SVMs
• Proximal Stochastic Dual Coordinate Ascent
• Distributed Stochastic Variance Reduced Gradient Methods and A Lower Bound for Communication Complexity
• A General Distributed Dual Coordinate Optimization Framework for Regularized Loss Minimization

Cited by in corpus (1)

• Accelerating Inexact Successive Quadratic Approximation for Regularized Optimization Through Manifold Identification