Stochastic quasi-Newton with adaptive step lengths for large-scale problems
arXiv:1802.04310
Abstract
We provide a numerically robust and fast method capable of exploiting the local geometry when solving large-scale stochastic optimisation problems. Our key innovation is an auxiliary variable construction coupled with an inverse Hessian approximation computed using a receding history of iterates and gradients. It is the Markov chain nature of the classic stochastic gradient algorithm that enables this development. The construction offers a mechanism for stochastic line search adapting the step length. We numerically evaluate and compare against current state-of-the-art with encouraging performance on real-world benchmark problems where the number of observations and unknowns is in the order of millions.
References in corpus (7)
- SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
- Stochastic Variance Reduction for Nonconvex Optimization
- RES: Regularized Stochastic BFGS Algorithm
- A Linearly-Convergent Stochastic L-BFGS Algorithm
- Global Convergence of Online Limited Memory BFGS
- Variance Reduction for Faster Non-Convex Optimization
- Stochastic Quasi-Newton Langevin Monte Carlo
Cited by in corpus (4)
- Gradient-only line searches: An Alternative to Probabilistic Line Searches
- Stochastic quasi-Newton with line-search regularization
- Gradient-only line searches to automatically determine learning rates for a variety of stochastic training algorithms
- Correlated pseudo-marginal Metropolis-Hastings using quasi-Newton proposals