An Accelerated Proximal Coordinate Gradient Method and its Application to Regularized Empirical Risk Minimization

Abstract

We consider the problem of minimizing the sum of two convex functions: one is smooth and given by a gradient oracle, and the other is separable over blocks of coordinates and has a simple known structure over each block. We develop an accelerated randomized proximal coordinate gradient (APCG) method for minimizing such convex composite functions. For strongly convex functions, our method achieves faster linear convergence rates than existing randomized proximal coordinate gradient methods. Without strong convexity, our method enjoys accelerated sublinear convergence rates. We show how to apply the APCG method to solve the regularized empirical risk minimization (ERM) problem, and devise efficient implementations that avoid full-dimensional vector operations. For ill-conditioned ERM problems, our method obtains improved convergence rates than the state-of-the-art stochastic dual coordinate ascent (SDCA) method.

References in corpus (12)

• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• Parallel Coordinate Descent for L1-Regularized Loss Minimization
• An Asynchronous Parallel Stochastic Coordinate Descent Algorithm
• Parallel Coordinate Descent Methods for Big Data Optimization
• Accelerated, Parallel and Proximal Coordinate Descent
• Iteration Complexity Analysis of Block Coordinate Descent Methods
• A Proximal Stochastic Gradient Method with Progressive Variance Reduction
• On the Complexity Analysis of Randomized Block-Coordinate Descent Methods
• Accelerated Proximal Stochastic Dual Coordinate Ascent for Regularized Loss Minimization
• A Randomized Nonmonotone Block Proximal Gradient Method for a Class of Structured Nonlinear Programming
• An Accelerated Randomized Kaczmarz Algorithm
• Efficient random coordinate descent algorithms for large-scale structured nonconvex optimization

Cited by in corpus (16)

• Variance Reduction for Faster Non-Convex Optimization
• Stochastic Primal-Dual Coordinate Method for Regularized Empirical Risk Minimization
• Randomized Dual Coordinate Ascent with Arbitrary Sampling
• SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimization
• An optimal randomized incremental gradient method
• A Unified Algorithmic Framework for Block-Structured Optimization Involving Big Data
• Coordinate Descent with Arbitrary Sampling I: Algorithms and Complexity
• Stochastic Dual Coordinate Ascent with Adaptive Probabilities
• An Accelerated Directional Derivative Method for Smooth Stochastic Convex Optimization
• Importance Sampling for Minibatches
• Worst-case Complexity of Cyclic Coordinate Descent: $O(n^2)$ Gap with Randomized Version
• Coordinate Descent with Arbitrary Sampling II: Expected Separable Overapproximation
• Primal-Dual Rates and Certificates
• Coordinate Descent Algorithms
• Numerical methods in large-scale optimization: inexact oracle and primal-dual analysis
• An Efficient Inexact ABCD Method for Least Squares Semidefinite Programming