Rethinking Gauss-Newton for learning over-parameterized models

Abstract

This work studies the global convergence and implicit bias of Gauss Newton's (GN) when optimizing over-parameterized one-hidden layer networks in the mean-field regime. We first establish a global convergence result for GN in the continuous-time limit exhibiting a faster convergence rate compared to GD due to improved conditioning. We then perform an empirical study on a synthetic regression task to investigate the implicit bias of GN's method. While GN is consistently faster than GD in finding a global optimum, the learned model generalizes well on test data when starting from random initial weights with a small variance and using a small step size to slow down convergence. Specifically, our study shows that such a setting results in a hidden learning phenomenon, where the dynamics are able to recover features with good generalization properties despite the model having sub-optimal training and test performances due to an under-optimized linear layer. This study exhibits a trade-off between the convergence speed of GN and the generalization ability of the learned solution.

References in corpus (19)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• New insights and perspectives on the natural gradient method
• Gradient Descent Finds Global Minima of Deep Neural Networks
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• Practical Gauss-Newton Optimisation for Deep Learning
• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networks
• A Unifying View on Implicit Bias in Training Linear Neural Networks
• Kernelized Wasserstein Natural Gradient
• Regularized Newton Method with Global $O(1/k^2)$ Convergence
• SGD with Large Step Sizes Learns Sparse Features
• Convergence beyond the over-parameterized regime using Rayleigh quotients
• Non-Convex Bilevel Games with Critical Point Selection Maps
• Flexible Modification of Gauss-Newton Method and Its Stochastic Extension
• On Feature Learning in Neural Networks with Global Convergence Guarantees

Cited by in corpus (1)

• Exact Gauss-Newton Optimization for Training Deep Neural Networks