Reducing Runtime by Recycling Samples
arXiv:1602.02136
Abstract
Contrary to the situation with stochastic gradient descent, we argue that when using stochastic methods with variance reduction, such as SDCA, SAG or SVRG, as well as their variants, it could be beneficial to reuse previously used samples instead of fresh samples, even when fresh samples are available. We demonstrate this empirically for SDCA, SAG and SVRG, studying the optimal sample size one should use, and also uncover be-havior that suggests running SDCA for an integer number of epochs could be wasteful.
References in corpus (12)
- A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
- Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
- A Universal Catalyst for First-Order Optimization
- Why Random Reshuffling Beats Stochastic Gradient Descent
- Even Faster Accelerated Coordinate Descent Using Non-Uniform Sampling
- Stochastic Primal-Dual Coordinate Method for Regularized Empirical Risk Minimization
- Stop Wasting My Gradients: Practical SVRG
- Finito: A Faster, Permutable Incremental Gradient Method for Big Data Problems
- Stochastic Optimization with Importance Sampling
- Randomized Dual Coordinate Ascent with Arbitrary Sampling
- Beneath the valley of the noncommutative arithmetic-geometric mean inequality: conjectures, case-studies, and consequences
- Stochastic Dual Coordinate Ascent with Adaptive Probabilities