The Dynamics of Gradient Descent for Overparametrized Neural Networks

Abstract

We consider the dynamics of gradient descent (GD) in overparameterized single hidden layer neural networks with a squared loss function. Recently, it has been shown that, under some conditions, the parameter values obtained using GD achieve zero training error and generalize well if the initial conditions are chosen appropriately. Here, through a Lyapunov analysis, we show that the dynamics of neural network weights under GD converge to a point which is close to the minimum norm solution subject to the condition that there is no training error when using the linear approximation to the neural network. To illustrate the application of this result, we show that the GD converges to a prediction function that generalizes well, thereby providing an alternative proof of the generalization results in Arora et al. (2019).

arXiv comment24 pages, published in parts at L4DC 2021

References in corpus (12)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Benign Overfitting in Linear Regression
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• On Lazy Training in Differentiable Programming
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Implicit Regularization in Deep Learning
• Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
• Risk and parameter convergence of logistic regression
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• Concentration Inequalities and Moment Bounds for Sample Covariance Operators
• Neural tangent kernels, transportation mappings, and universal approximation