Exact Gauss-Newton Optimization for Training Deep Neural Networks

Abstract

We present Exact Gauss-Newton (EGN), a stochastic second-order optimization algorithm that combines the generalized Gauss-Newton (GN) Hessian approximation with low-rank linear algebra to compute the descent direction. Leveraging the Duncan-Guttman matrix identity, the parameter update is obtained by factorizing a matrix which has the size of the mini-batch. This is particularly advantageous for large-scale machine learning problems where the dimension of the neural network parameter vector is several orders of magnitude larger than the batch size. Additionally, we show how improvements such as line search, adaptive regularization, and momentum can be seamlessly added to EGN to further accelerate the algorithm. Moreover, under mild assumptions, we prove that our algorithm converges in expectation to a stationary point of the objective. Finally, our numerical experiments demonstrate that EGN consistently exceeds, or at most matches the generalization performance of well-tuned SGD, Adam, GAF, SQN, and SGN optimizers across various supervised and reinforcement learning tasks.

Accepted to Neurocomputing (2025). This version corresponds to the accepted manuscript; minor formatting edits for arXiv single-column layout (34 pages, 3 figures, 8 tables)

References in corpus (4)

• Proceedings of the 29th International Conference on Machine Learning (ICML-12)
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• SCORE: Approximating Curvature Information under Self-Concordant Regularization
• Rethinking Gauss-Newton for learning over-parameterized models