Improved Optimization of Finite Sums with Minibatch Stochastic Variance Reduced Proximal Iterations
arXiv:1706.07001
Abstract
We present novel minibatch stochastic optimization methods for empirical risk minimization problems, the methods efficiently leverage variance reduced first-order and sub-sampled higher-order information to accelerate the convergence speed. For quadratic objectives, we prove improved iteration complexity over state-of-the-art under reasonable assumptions. We also provide empirical evidence of the advantages of our method compared to existing approaches in the literature.
References in corpus (13)
- SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
- Optimal Distributed Online Prediction using Mini-Batches
- Communication Efficient Distributed Optimization using an Approximate Newton-type Method
- Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting
- Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization
- A Linearly-Convergent Stochastic L-BFGS Algorithm
- Sub-sampled Newton Methods with Non-uniform Sampling
- Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk minimization
- Sub-Sampled Newton Methods I: Globally Convergent Algorithms
- Sub-Sampled Newton Methods II: Local Convergence Rates
- Tight Complexity Bounds for Optimizing Composite Objectives
- Second-Order Stochastic Optimization for Machine Learning in Linear Time
- Memory and Communication Efficient Distributed Stochastic Optimization with Minibatch-Prox
Cited by in corpus (7)
- Gradient Sparsification for Communication-Efficient Distributed Optimization
- First-order Newton-type Estimator for Distributed Estimation and Inference
- Simultaneous Inference for Massive Data: Distributed Bootstrap
- Distributed Bootstrap for Simultaneous Inference Under High Dimensionality
- Stochastic Variance Reduction for Policy Gradient Estimation
- Trajectory Normalized Gradients for Distributed Optimization
- Curvature-Exploiting Acceleration of Elastic Net Computations