Inexact Newton Methods for Stochastic Nonconvex Optimization with Applications to Neural Network Training

Abstract

We study stochastic inexact Newton methods and consider their application in nonconvex settings. Building on the work of [R. Bollapragada, R. H. Byrd, and J. Nocedal, IMA Journal of Numerical Analysis, 39 (2018), pp. 545--578] we derive bounds for convergence rates in expected value for stochastic low rank Newton methods, and stochastic inexact Newton Krylov methods. These bounds quantify the errors incurred in subsampling the Hessian and gradient, as well as in approximating the Newton linear solve, and in choosing regularization and step length parameters. We deploy these methods in training convolutional autoencoders for the MNIST and CIFAR10 data sets. Numerical results demonstrate that, relative to first order methods, these stochastic inexact Newton methods often converge faster, are more cost-effective, and generalize better.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• How to Escape Saddle Points Efficiently
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• When Are Nonconvex Problems Not Scary?
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• Sub-Sampled Newton Methods I: Globally Convergent Algorithms
• Sub-Sampled Newton Methods II: Local Convergence Rates
• Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information
• Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study
• A Generic Approach for Escaping Saddle points

Cited by in corpus (6)

• Derivative-Informed Projected Neural Networks for High-Dimensional Parametric Maps Governed by PDEs
• Fast Approximation of the Gauss-Newton Hessian Matrix for the Multilayer Perceptron
• Ill-Posedness and Optimization Geometry for Nonlinear Neural Network Training
• Low Rank Saddle Free Newton: A Scalable Method for Stochastic Nonconvex Optimization
• BEAR: Sketching BFGS Algorithm for Ultra-High Dimensional Feature Selection in Sublinear Memory
• Deep Neural Network Learning with Second-Order Optimizers -- a Practical Study with a Stochastic Quasi-Gauss-Newton Method