Stochastic quasi-Newton with line-search regularization

Abstract

In this paper we present a novel quasi-Newton algorithm for use in stochastic optimisation. Quasi-Newton methods have had an enormous impact on deterministic optimisation problems because they afford rapid convergence and computationally attractive algorithms. In essence, this is achieved by learning the second-order (Hessian) information based on observing first-order gradients. We extend these ideas to the stochastic setting by employing a highly flexible model for the Hessian and infer its value based on observing noisy gradients. In addition, we propose a stochastic counterpart to standard line-search procedures and demonstrate the utility of this combination on maximum likelihood identification for general nonlinear state space models.

References in corpus (3)

• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• Stochastic (Approximate) Proximal Point Methods: Convergence, Optimality, and Adaptivity
• Evaluating the squared-exponential covariance function in Gaussian processes with integral observations

Cited by in corpus (4)

• Gradient-only line searches: An Alternative to Probabilistic Line Searches
• Nonlinear state-space identification using deep encoder networks
• Learning the Step-size Policy for the Limited-Memory Broyden-Fletcher-Goldfarb-Shanno Algorithm
• Gradient-only line searches to automatically determine learning rates for a variety of stochastic training algorithms