A Stochastic Quasi-Newton Method with Nesterov's Accelerated Gradient

Abstract

Incorporating second order curvature information in gradient based methods have shown to improve convergence drastically despite its computational intensity. In this paper, we propose a stochastic (online) quasi-Newton method with Nesterov's accelerated gradient in both its full and limited memory forms for solving large scale non-convex optimization problems in neural networks. The performance of the proposed algorithm is evaluated in Tensorflow on benchmark classification and regression problems. The results show improved performance compared to the classical second order oBFGS and oLBFGS methods and popular first order stochastic methods such as SGD and Adam. The performance with different momentum rates and batch sizes have also been illustrated.

arXiv commentAccepted at ECML-PKDD 2019

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• RES: Regularized Stochastic BFGS Algorithm
• A Linearly-Convergent Stochastic L-BFGS Algorithm
• Global Convergence of Online Limited Memory BFGS
• Sub-Sampled Newton Methods I: Globally Convergent Algorithms
• A Progressive Batching L-BFGS Method for Machine Learning
• A Variance Reduced Stochastic Newton Method
• Implementation of a modified Nesterov's Accelerated quasi-Newton Method on Tensorflow
• Implementation of Stochastic Quasi-Newton's Method in PyTorch

Cited by in corpus (1)

• A Stochastic Variance Reduced Nesterov's Accelerated Quasi-Newton Method