Practical Quasi-Newton Methods for Training Deep Neural Networks

Abstract

We consider the development of practical stochastic quasi-Newton, and in particular Kronecker-factored block-diagonal BFGS and L-BFGS methods, for training deep neural networks (DNNs). In DNN training, the number of variables and components of the gradient $n$ is often of the order of tens of millions and the Hessian has $n^2$ elements. Consequently, computing and storing a full $n\times n$ BFGS approximation or storing a modest number of (step, change in gradient) vector pairs for use in an L-BFGS implementation is out of the question. In our proposed methods, we approximate the Hessian by a block-diagonal matrix and use the structure of the gradient and Hessian to further approximate these blocks, each of which corresponds to a layer, as the Kronecker product of two much smaller matrices. This is analogous to the approach in KFAC, which computes a Kronecker-factored block-diagonal approximation to the Fisher matrix in a stochastic natural gradient method. Because the indefinite and highly variable nature of the Hessian in a DNN, we also propose a new damping approach to keep the upper as well as the lower bounds of the BFGS and L-BFGS approximations bounded. In tests on autoencoder feed-forward neural network models with either nine or thirteen layers applied to three datasets, our methods outperformed or performed comparably to KFAC and state-of-the-art first-order stochastic methods.

References in corpus (13)

• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• RES: Regularized Stochastic BFGS Algorithm
• A Linearly-Convergent Stochastic L-BFGS Algorithm
• Global Convergence of Online Limited Memory BFGS
• Krylov Subspace Descent for Deep Learning
• Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
• Natural Neural Networks
• Shampoo: Preconditioned Stochastic Tensor Optimization
• Practical Gauss-Newton Optimisation for Deep Learning
• A Variance Reduced Stochastic Newton Method
• Information Newton's flow: second-order optimization method in probability space
• A Neural Network model with Bidirectional Whitening
• Modular Block-diagonal Curvature Approximations for Feedforward Architectures

Cited by in corpus (10)

• Scalable Second Order Optimization for Deep Learning
• KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks
• Bayesian inverse Navier-Stokes problems: joint flow field reconstruction and parameter learning
• Analytic Insights into Structure and Rank of Neural Network Hessian Maps
• LocoProp: Enhancing BackProp via Local Loss Optimization
• NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation
• NG+ : A Multi-Step Matrix-Product Natural Gradient Method for Deep Learning
• A Trace-restricted Kronecker-Factored Approximation to Natural Gradient
• Enhance Curvature Information by Structured Stochastic Quasi-Newton Methods
• mL-BFGS: A Momentum-based L-BFGS for Distributed Large-Scale Neural Network Optimization