Enhance Curvature Information by Structured Stochastic Quasi-Newton Methods
arXiv:2006.09606
Abstract
In this paper, we consider stochastic second-order methods for minimizing a finite summation of nonconvex functions. One important key is to find an ingenious but cheap scheme to incorporate local curvature information. Since the true Hessian matrix is often a combination of a cheap part and an expensive part, we propose a structured stochastic quasi-Newton method by using partial Hessian information as much as possible. By further exploiting either the low-rank structure or the kronecker-product properties of the quasi-Newton approximations, the computation of the quasi-Newton direction is affordable. Global convergence to stationary point and local superlinear convergence rate are established under some mild assumptions. Numerical results on logistic regression, deep autoencoder networks and deep convolutional neural networks show that our proposed method is quite competitive to the state-of-the-art methods.
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Deep Learning in Neural Networks: An Overview
- Language Models are Few-Shot Learners
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Practical Quasi-Newton Methods for Training Deep Neural Networks
- Practical Gauss-Newton Optimisation for Deep Learning
- A Multi-Batch L-BFGS Method for Machine Learning
- Structured Quasi-Newton Methods for Optimization with Orthogonality Constraints
- Sketchy Empirical Natural Gradient Methods for Deep Learning
- A Fully Stochastic Second-Order Trust Region Method