An Accelerated Proximal Coordinate Gradient Method and its Application to Regularized Empirical Risk Minimization

Abstract

We consider the problem of minimizing the sum of two convex functions: one is smooth and given by a gradient oracle, and the other is separable over blocks of coordinates and has a simple known structure over each block. We develop an accelerated randomized proximal coordinate gradient (APCG) method for minimizing such convex composite functions. For strongly convex functions, our method achieves faster linear convergence rates than existing randomized proximal coordinate gradient methods. Without strong convexity, our method enjoys accelerated sublinear convergence rates. We show how to apply the APCG method to solve the regularized empirical risk minimization (ERM) problem, and devise efficient implementations that avoid full-dimensional vector operations. For ill-conditioned ERM problems, our method obtains improved convergence rates than the state-of-the-art stochastic dual coordinate ascent (SDCA) method.

References in corpus (3)

• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• Parallel Coordinate Descent for L1-Regularized Loss Minimization
• A Proximal Stochastic Gradient Method with Progressive Variance Reduction

Cited by in corpus (7)

• Randomized Dual Coordinate Ascent with Arbitrary Sampling
• SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimization
• Coordinate Descent with Arbitrary Sampling I: Algorithms and Complexity
• Stochastic Dual Coordinate Ascent with Adaptive Probabilities
• Coordinate Descent with Arbitrary Sampling II: Expected Separable Overapproximation
• Coordinate Descent Algorithms
• An Efficient Inexact ABCD Method for Least Squares Semidefinite Programming