A Stochastic Quasi-Newton Method for Large-Scale Optimization

Abstract

The question of how to incorporate curvature information in stochastic approximation methods is challenging. The direct application of classical quasi- Newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration. In this paper, we propose a stochastic quasi-Newton method that is efficient, robust and scalable. It employs the classical BFGS update formula in its limited memory form, and is based on the observation that it is beneficial to collect curvature information pointwise, and at regular intervals, through (sub-sampled) Hessian-vector products. This technique differs from the classical approach that would compute differences of gradients, and where controlling the quality of the curvature estimates can be difficult. We present numerical results on problems arising in machine learning that suggest that the proposed method shows much promise.

Cited by in corpus (11)

• Convergence rates of sub-sampled Newton methods
• On the properties of variational approximations of Gibbs posteriors
• SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimization
• Second-Order Stochastic Optimization for Machine Learning in Linear Time
• A globally convergent incremental Newton method
• Stochastic Quasi-Newton Langevin Monte Carlo
• A Variance Reduced Stochastic Newton Method
• A Proximal Stochastic Quasi-Newton Algorithm
• Newton-Stein Method: An optimization method for GLMs via Stein's Lemma
• Full waveform inversion with random shot selection using adaptive gradient descent
• Accelerated Stochastic Quasi-Newton Optimization on Riemann Manifolds