Improved Optimization of Finite Sums with Minibatch Stochastic Variance Reduced Proximal Iterations

Abstract

We present novel minibatch stochastic optimization methods for empirical risk minimization problems, the methods efficiently leverage variance reduced first-order and sub-sampled higher-order information to accelerate the convergence speed. For quadratic objectives, we prove improved iteration complexity over state-of-the-art under reasonable assumptions. We also provide empirical evidence of the advantages of our method compared to existing approaches in the literature.

References in corpus (3)

• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• Sub-sampled Newton Methods with Non-uniform Sampling
• Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk minimization

Cited by in corpus (3)

• Simultaneous Inference for Massive Data: Distributed Bootstrap
• Trajectory Normalized Gradients for Distributed Optimization
• Curvature-Exploiting Acceleration of Elastic Net Computations