SGD with Variance Reduction beyond Empirical Risk Minimization
arXiv:1510.04822
Abstract
We introduce a doubly stochastic proximal gradient algorithm for optimizing a finite average of smooth convex functions, whose gradients depend on numerically expensive expectations. Our main motivation is the acceleration of the optimization of the regularized Cox partial-likelihood (the core model used in survival analysis), but our algorithm can be used in different settings as well. The proposed algorithm is doubly stochastic in the sense that gradient steps are done using stochastic gradient descent (SGD) with variance reduction, where the inner expectations are approximated by a Monte-Carlo Markov-Chain (MCMC) algorithm. We derive conditions on the MCMC number of iterations guaranteeing convergence, and obtain a linear rate of convergence under strong convexity and a sublinear rate without this assumption. We illustrate the fact that our algorithm improves the state-of-the-art solver for regularized Cox partial-likelihood on several datasets from survival analysis.
arXiv comment17 pages
References in corpus (7)
- SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
- Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting
- Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
- Proximal Stochastic Dual Coordinate Ascent
- Stop Wasting My Gradients: Practical SVRG
- Scalable Kernel Methods via Doubly Stochastic Gradients
- Less than a Single Pass: Stochastically Controlled Stochastic Gradient Method