A Proximal Stochastic Quasi-Newton Algorithm

Abstract

In this paper, we discuss the problem of minimizing the sum of two convex functions: a smooth function plus a non-smooth function. Further, the smooth part can be expressed by the average of a large number of smooth component functions, and the non-smooth part is equipped with a simple proximal mapping. We propose a proximal stochastic second-order method, which is efficient and scalable. It incorporates the Hessian in the smooth part of the function and exploits multistage scheme to reduce the variance of the stochastic gradient. We prove that our method can achieve linear rate of convergence.

References in corpus (6)

• Templates for Convex Cone Problems with Applications to Sparse Signal Recovery
• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• A Linearly-Convergent Stochastic L-BFGS Algorithm
• A quasi-Newton proximal splitting method
• Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods
• A Stochastic Quasi-Newton Method for Large-Scale Optimization

Cited by in corpus (3)

• Stochastic L-BFGS: Improved Convergence Rates and Practical Acceleration Strategies
• Stochastic Newton and Cubic Newton Methods with Simple Local Linear-Quadratic Rates
• Training L1-Regularized Models with Orthant-Wise Passive Descent Algorithms